openai-php / client

⚡️ OpenAI PHP is a supercharged community-maintained PHP API client that allows you to interact with OpenAI API.
MIT License
4.56k stars 465 forks source link

[Bug]: Faking custom metadata key-value pairs in responses is not possible #409

Closed Radiergummi closed 1 month ago

Radiergummi commented 1 month ago

Description

We use custom metadata attributes (e.g. these) extensively to automatically manage resource revisions. For example, we create assistants with a checksum metadata property to verify a given assistant matches the definition in our source code to detect changes via the playground, and recreate the assistant in these cases, to avoid undefined behaviour in production.

Therefore, in test cases, I would like to be able to ensure a given code path will correctly check for the existence of an assistant; to do so, I fake an AssistantListResponse, and include the metadata key with the test checksum.

The response returned, however, will not include the metadata property. On further inspection, I discovered faking any metadata properties is not possible, due to the way the Fakeable::buildAttributes() function works. When encountering additional properties not defined in response fixtures, it will add them to the final response payload, but only if those properties are numeric. A side effect of that, however, is that metadata cannot be faked at all currently, despite it is supposed to be a dictionary of user-defined key-value pairs.

Considering the aforementioned numeric key check, which probably exists for a reason, should stay in place: Maybe a special exception could be made for metadata? That property exists on pretty much all OpenAI API resources, and is the only dictionary property I'm aware of. This could be as simple as:

  foreach ($override as $key => $value) {
-     if (! is_numeric($key)) {
+     if (! is_numeric($key) && $key !== 'metadata') {
          continue;
      }

      $new[$key] = $value;
  }

Steps To Reproduce

  1. Fake a response, for example:
    $fakeAssistants = AssistantListResponse::fake([
       'data' => [ [
           'metadata' => [ 'foo' => 'bar' ]
       ] ]
    ]);
  2. Verify the metadata array is empty:
    assert(
       !empty($fakeAssistants->data[0]->metadata),
       'Metadata is empty, despite "foo" being included in the fake call above'
    );

OpenAI PHP Client Version

v0.9.0

PHP Version

8.3

Notes

No response

gehrisandro commented 1 month ago

Hi @Radiergummi

Thanks for raising the issue and your comprehensive explanation. Issue fixed in https://github.com/openai-php/client/pull/414