benjmor
commented
8 months ago One option is going to be pulling the prompts in advance and passing them to ChatGPT at request-time.
It's a little bit slower for individuals (the full tournament doesn't get results generated), but it's a lot cheaper and could operate synchronously.
It's going to be tricky to find a balance here!
In a world without resource constraints, this would run as a batch job overnight against all tournaments that completed the previous day.
However, the ad-hoc runs are necessary currently to balance cost vs who actually wants these results.