Azure / azure-openai-benchmark

Azure OpenAI benchmarking tool
MIT License
130 stars 55 forks source link

Add 'replay' context generation method #17

Closed michaeltremeer closed 6 months ago

michaeltremeer commented 10 months ago

This PR adds a second method for generating messages, so that users can load a list of messages payloads from file and use these for testing (which a number of customers are looking to use for real-world testing).

This PR:

technicianted commented 10 months ago

The PR is about replay but then we're just using the dataset randomly which would not give the same usage patterns as the original dataset. For example, if I provide a dataset that has cyclical patterns in them that represent my shape over 4 hours for example, the results are not going to be what's expected.

Please correct me if I'm misunderstanding the use-case here.

I understand that this may add another layer of complexity: what about timing of the dataset? Originally I had thought that this was going to be your idea: in replay mode we don't provide RPM. Instead, we get timing from the dataset itself and we replay it the same way.

One proposal is to keep this as-is but name it something different and use the dataset sequentially rather than in random. Perhaps call it file. Then we can work on a completely different subcommand that would implement a true dataset replay.

What do you think?