search

ethz-spylab / satml-llm-ctf

Code used to run the platform for the LLM CTF colocated with SaTML 2024
https://ctf.spylab.ai
MIT License
23 stars 5 forks source link

OpenAI timeout when using Llama models #59

Closed epistoteles closed 7 months ago

epistoteles commented 7 months ago

The API is under heavy load right now. I'm trying to work on some Llama models.

However, it seems the API is doing some OpenAI requests in the background? At least I'm getting a 60 second OpenAI timeout error message.

Could you look into this and potentially remove OpenAI requests from Llama chats? This might free up some capacity.

dedeswim commented 7 months ago

From the Google Group announcement from yesterday:

Our TogetherAI and OpenAI accounts have rate limits; there’s nothing we can do about this. If you start getting OpenAI errors (this is the error it will raise for both models, as we’re indirectly using openai-python for both) using the team budget tomorrow, try using your own keys.

epistoteles commented 7 months ago

Alright, thanks for the quick response!

dpaleka commented 7 months ago

It seems TogetherAI upstream API is having issues. Can some participants try if it's also extremely slow for them too, on their personal keys?

If this is the root cause, there is really nothing we can do about this, except decrease \beta. We'll do an announcement shortly.

s-zanella commented 7 months ago

I confirm that Together API is also unresponsive using paid tier keys.

dedeswim commented 7 months ago

We released an announcement. Please rereport any errors that are not covered by the announcement, thanks!

dedeswim commented 7 months ago

Looks like their API is up again

dedeswim commented 7 months ago

Looks like Together's API is having some issues again. We'll monitor the situation and act if the problem does not get solved soon.

dedeswim commented 7 months ago

Looks like the API is up again, closing but keeping pinned.

Sandr0x00 commented 6 months ago

lol together ... of course 2h before the end :)

dedeswim commented 6 months ago

It seems to be up again?

Sandr0x00 commented 6 months ago

Yep, only was down like 20-30min or so, luckily got up again :)