MaartenGr
commented
5 months ago Thank you for the extensive description! Indeed, test coverage is quite tricky at the moment especially with a key-based external provider.
With respect to your suggestions, the openai-responses package looks great but is not maintained by OpenAI itself nor has the popularity/usage where I am confident it will be maintained in the upcoming years. Since we are going to be dependent on that package, it is important that we can be relatively sure it will continue development. Moreover, it introduces another package with dependencies that might result in dependency conflicts.
I don't have much experience with MagicMock, but I can imagine the difficulty of setting something up from scratch.
steven-solomon
Goal
I would like to make it easier to verify the behavior of the
OpenAIclass by adding test coverage.Proposal
Test coverage will aim at the following scenarios—both for their happy path and edge cases. Although, additional tests may be added in the implementation phase.
Cases to cover:
Tests will be divided into happy path and edge case scenarios to make nominal and exceptional behaviors clear to the reader. In order to hold the design of the
OpenAIclass static, I would like to pursue a mocking strategy for theopenai.OpenAI.client. I would like to create two drafts with one using OpenAI-Responses package and the other using unittest.mock.MagicMock. I will defer to your preference on readability between the two libraries.Considerations
Here are some tradeoffs that I considered when drafting this proposal:
The inherent risk of mock based testing is that the underlying API can diverge from the mock's behavior. However, it allows us to have confidence of the integration on the edges of our system.
When selecting a mocking strategy, I considered using either a built in Python package or a specialized package that provides a simpler interface. The OpenAI-Responses package can make it easy to mock the OpenAI behavior, but in the scenario of an API functionality drifts, the update loop may be longer or completely blocked by the maintainer of that package. In terms of the built in
MagicMockclass, mocking will be verbose and difficult to reason about, coupling your codebase to the nested attributes of the API client. Comprehension can be improved using construction patterns but that adds to the maintenance cost.Despite the downsides, having an additional automated quality check outside of manual testing can make both contributions and reviews easier.