search

explodinggradients / ragas

Supercharge Your LLM Application Evaluations 🚀
https://docs.ragas.io
Apache License 2.0
7.3k stars 746 forks source link

Timeout Error #1667

Open Dormiveglia-elf opened 1 week ago

Dormiveglia-elf commented 1 week ago

ERROR:ragas.executor:Exception raised in Job[2]: TimeoutError() ERROR:ragas.executor:Exception raised in Job[7]: TimeoutError() ERROR:ragas.executor:Exception raised in Job[10]: TimeoutError() ERROR:ragas.executor:Exception raised in Job[15]: TimeoutError() ....

I use GPT-4o and encountered many timeout error, any ideas?

Dormiveglia-elf commented 1 week ago

I have reduced the number of timeout error by adjusting the max_workers to 60 and time_out parameter to 120 mentioned in https://docs.ragas.io/en/stable/howtos/customizations/_run_config/?h=time

But still some error existing, I think 120 is a very long time. Any ideas?

YuffieHuang commented 1 week ago

max_workers

Further reducing max_workers solved the same issue for me. I set it to be 8 for my locally hosted models.

jjmachan commented 1 week ago

hey anothe approach is batch_size argument for evalaute() https://docs.ragas.io/en/stable/references/evaluate/#ragas.evaluation.evaluate

something else you could try? easier to reason about IMO