[Bug]: Sanitize OpenAI input

[tazarov]

What happened?

If texts passed to OpenAI API are breaking the JSON our wrapper sends 400 error is returned. Maybe there is a way we can sanitize output or leave it to openai lib to fix that.

https://discord.com/channels/1073293645303795742/1183798062863372318

Versions

Any

Relevant log output

ERROR:root:Error: Error code: 400 - {'error': {'message': "'$.input' is invalid. Please check the API reference: https://platform.openai.com/docs/api-reference.", 'type': 'invalid_request_error', 'param': None, 'code': None}}
Traceback (most recent call last):
  File "/home/matischroder/sanatorio_allende/upload_web_chroma.py", line 94, in <module>
    main()
  File "/home/matischroder/sanatorio_allende/upload_web_chroma.py", line 87, in main
    collection_id = train_with_url(
  File "/home/matischroder/sanatorio_allende/upload_web_chroma.py", line 77, in train_with_url
    raise e
  File "/home/matischroder/sanatorio_allende/upload_web_chroma.py", line 66, in train_with_url
    chroma_collection.add(
  File "/home/matischroder/sanatorio_allende/venv/lib/python3.9/site-packages/chromadb/api/models/Collection.py", line 147, in add
    embeddings = self._embed(input=documents)

[HammadB]

For my understanding - What are the cases where the json serialization in openais lib (assuming this is coming from there) breaks?

[tazarov]

According to elmatero in discord:

Going to test it out and see if reproducible.

[beggers]

I'm a little confused -- is this an issue with the OpenAI API itself, with OpenAI's python client, or with our EmbeddingFunction which wraps OpenAI's python library?

[HammadB]

I think OpenAI will throw an error if given None, is that correct @tazarov ?