ronaldmannak
commented
1 year ago FYI: The previous implementation was probably fine for 95%+ of the use cases. The memory leak only occurred when releasing openAIKit while the app is running. Most implementations probably never released the client until the app shuts downs.
One thing I've been thinking of is the possibility of overriding the default URL in case you want to use a proxy. With that, an example AWS Lambda implementation would be good to include. However, for GPT4 we'd really need streaming. GPT4 is too slow and AWS Gateway will timeout. Maybe we could add a AWS lambda example, restructure the HTTP client and implement streaming next.
I don't have any strong opinion about how to do that and I'm not super familiar with Nio. Is your suggestion to use URLSession instead of Nio or can URLSession be used on top of Nio?
dylanshine
@ronaldmannak I was thinking based on your last contribution, that it might make sense to restructure the use of AsyncHTTPClient to clean up usage. I was also thinking it might make sense to potentially allow passing either an AsyncHTTPClient or URLSession in order to perform the requests. Haven't really sat down to think this through, but I'd be curious to hear your thoughts.