Simultaneous submissions cause slow reponses

[ff-dh]

If multiple URLs are submitted at roughly the same time, the later submissions take longer to render while the load is increased.

Example (with one worker):

• Users 1 through 10 each submit a URL at 12:00. The POST requests happen sequentially because they're queued up and handled when /post route is called, which returns a stream_template.
• User 1 gets an immediate response.
• User 2 also maybe gets an immediate response.
• User 3 may get an almost-immediate response. Claims will be coming through (albeit slowly) for these users.
• Users 4-10 will see nothing except their browser spinning until one of those analyses are complete or there's an exception raised. These may timeout on the user's end, or because of a proxy, or whatever.

I thought the database might've been blocking these but after some db changes for throughput and reduced blocking this particular issue wasn't resolved. It could be a stream_template upper limit but the docs don't speak on anything like that.

[andylolz]

Yes, this probably does relate to using stream_template.

We currently run gunicorn with 2 synchronous workers: https://github.com/FullFact/health-misinfo-shared/blob/ae033e995f323e5d5ed3207e350c0980b593d255/Dockerfile#L21

We could up the number of workers, and/or make them async.

[andylolz]

It’s probably worth writing down somewhere: stream_template is a massive hack here, to avoid running the LLM in background tasks in celery. It would be much better to use background tasks here, but also a load of faff that is probably not worth it for an MVP.