Multiple calls to get_rate_limits in OpenAIService slows down the exectime of surveys when using cache.

[zer0dss]

This code uses a cache to run the survey. The first call to survey run takes ~2s; for the remaining 99 calls in normal cases, the results should be obtained instantly from the cache, but it takes ~1s.

from edsl import QuestionFreeText,QuestionMultipleChoice
from edsl import Cache,Survey
import time

s = Survey(questions=[QuestionFreeText.example(),QuestionMultipleChoice.example()])
c = Cache()

for i in range(0,100):
    start = time.time()
    res = s.run(cache=c)
    print("exec time ",time.time()-start)

[johnjosephhorton]

Good catch - Maybe we just put an LRU cache decorator on the limit lookup function for now?

On Fri, Jun 28, 2024 at 1:57 PM stefanm @.***> wrote:

This code uses a cache to run the survey. The first call to survey run takes ~2s; for the remaining 99 calls in normal cases, the results should be obtained instantly from the cache, but it takes ~1s.

from edsl import QuestionFreeText,QuestionMultipleChoicefrom edsl import Cache,Surveyimport time s = Survey(questions=[QuestionFreeText.example(),QuestionMultipleChoice.example()])c = Cache() for i in range(0,100): start = time.time() res = s.run(cache=c) print("exec time ",time.time()-start)

— Reply to this email directly, view it on GitHub https://github.com/expectedparrot/edsl/issues/699, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADDNMTWDU56UBQJXQQFZG3ZJWP2HAVCNFSM6AAAAABKCJRQCCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM4DAOJRGMZTGMQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[johnjosephhorton]

@zer0dss I think a good solution here would be:

1. we keep alternative/back up lists of services in models hard-coded in edsl/enums.py that we can use
2. we create a warning of some kind to (us) as developers when a API model lookup reveals a discrepancy between our backup list (as a sign we need to update)
3. we don't throw an exception when we can't access the API list but we do print a warning for the user that we're using a stored list

Thoughts on this approach?

[zer0dss]

@johnjosephhorton This is the solution that I was thinking too. Until now I think this is the only place where we have this type of requests that slows down the execution ( checking for available models)

[johnjosephhorton]

Ok - excellent that we're aligned. Do you want to build it?

[zer0dss]

Yes, I will add it.

[johnjosephhorton]

Hey @zer0dss not to pressure you, but do you have an ETA on this one? It's holding up some stuff I want to do on this new 'Study' module.

[zer0dss]

@johnjosephhorton Currently I am working on a coop task. I can do a fast patch today that will cache the values (a similar patch to the one applied to the method that checks for available models). How is this blocking the work on the Study module, do you want to push a new edsl version that has this running optimization in place?

[johnjosephhorton]

No worries - I'll do a quick fix myself.

re: blocking - so I'm trying to debug some evidence that a study is making an rogue API call on a replication when calls should purely be from the cache. And the most error-free way to check is to turn off wifi :)

[zer0dss]

Ok, it makes sense to me now why this blocks your work on the study module.