Open EdIzaguirre opened 1 month ago
Hey! Thanks for writing in, let's get to the bottom of this. What version of weave are you running? We recently made changes relating to parallelism in the weave library, upgrading might be worth a shot. Also, you say that everything works 80% of the time, i'm assuming that those 20% runtime error occur non-deterministically?
I am seeing notes internally that we have run into this issue with Ragas around a month ago, but interestingly similar rag scoring frameworks like Tonic worked fine. You might also be interested to know that an integration with ragas
is on our roadmap.
Internal ticket for tracking.
I am using weave==0.50.10
, the latest one. Correct, the error occurs non-deterministically. I may have to check out the Tonic library.
Perfect thanks, this issue has been triaged and has been added to our backlog. A deeper investigation is imminent, but because the issue is an interaction with an external library it can't be prioritized over other internal issues. We will keep you posted with progress, thanks again for writing this up!
I've been enjoying the Weave library quite a bit, but I have been running into an issue using the Evaluate method. The issue is that 20% of the time, when running my evaluation, I get the
Runtime Error
mentioned below referring to the use of different event loops. 80% of the time my evaluation is performed as normal. The thing is I don't see where the second event loop is being called? Part of the complication is that I am using Ragas to evaluate a few metrics for my chatbot, and I don't want to modify the Ragas library to accomplish my goal. I have tried following both of the examples given in the tutorials, but they both do it in slightly different ways. For the record, addingasync
to theevaluate_with_ragas
method did not work, nor did replacinginvoke
withainvoke
and usingawait
in mypredict
method. I have already limitedWEAVE_PARALLELISM
to stem any issues from rate limiting. Any help would be appreciated!Code
_evaluatemodel.py
Relevant portion of FilmSearchModel.py:
Error: