search

assafelovic / gpt-researcher

GPT based autonomous agent that does online comprehensive research on any given topic
https://gptr.dev
MIT License
12.98k stars 1.61k forks source link

Support disabling of API response streaming via config #591

Open barsuna opened 2 weeks ago

barsuna commented 2 weeks ago

When using obscure LLMs/APIs that do not do streaming, it seems it is not possible for API just to return results in 1 go - there are 2 places in actions.py where streaming is hardcoded. It is possible to move this to config so that one could disable streaming without editing code?

assafelovic commented 2 weeks ago

Hey @barsuna can you describe the reasoning behind why you'd want to disable streaming?

barsuna commented 2 weeks ago

Hey @assafelovic, it is hardly rational i know - it is an 'operating in a broken world' kind of thing. I found some odd issues where APIs wouldn't handle correctly the streaming (sometimes they quietly ignore it and operate as if stream=false, sometimes they try and stream, but that is broken in itself). What i found is that we dont immediatelly see the error in gpt-researcher at the point of API call. Rather the system keeps going and fails at later stage (usually with empty reports) - made troubleshooting rather lengthy. This is not high-priority for me, just sharing the lessons learned.

The right thing is of course to go and fix the APIs - but in some cases that is lengthy, improbable or impossible - IMO it helps adoption if users have a workaround.