Support UTF-8 characters (e.g. Hebrew)

[orenagiv]

Hey, While using the LangChain.dart package, which uses the openapi_spec package, I've noticed that if the OpenAI response includes characters in Hebrew or other languages with special characters - it results with "gibberish".

This is most probably due to client.dart > lines 883 and 896, where json.decode is done over r.body: https://github.com/search?q=repo%3Atazatechnology%2Fopenapi_spec+json.decode%28r.body%29&type=code

It seems that due to this Dart issue: https://github.com/dart-lang/http/issues/789

The json.decode should be done with utf8.decode over the r.bodyBytes, in order to properly handle the UTF-8 chars:

if (s.ref != null || returnType.startsWith('Union')) {
  decoder = "return $returnType.fromJson(json.decode(utf8.decode(r.bodyBytes)));";
} else {
  decoder = """
    final list = json.decode(utf8.decode(r.bodyBytes)) as List;
    return list.map((e) => ${s.items.ref}.fromJson(e)).toList();

I've manually manipulated the generated client.dart file - and it resolved the problem. However, unfortunately, due to lack of time I'm unable to test this properly and submit a Pull Request... I hope the maintainers will have to take a look soon.

[walsha2]

Looks like there is some confusion in the http package about the expected/default decode logic. I assumed that utf-8 was the default, but I suppose that is not the case.

@orenagiv this has been taken care of in #44 and successfully published to openapi_spec (v0.7.8). @davidmigloz should be able to integrate in to LangChain Dart soon.

[davidmigloz]

Fixed in:

• openai_dart: 0.1.1+2
• langchain_openai 0.1.2+1

Thank you both!

[davidmigloz]

@walsha2 we need to recreate the other clients as well (pinecone on your side and chromadb on my side).

[davidmigloz]

chromadb recreated in:

• chromadb: 0.1.0+2
• langchain_chroma: 0.1.0+5

[orenagiv]

Thanks @davidmigloz @walsha2 ! Works great! 👍

[walsha2]

@walsha2 we need to recreate the other clients as well (pinecone on your side and chromadb on my side).

• pinecone: 0.7.2