leondz
commented
6 months ago - [ ] @leondz Write a test for this
Closed leondz closed 6 months ago
leondz
commented
6 months ago 
jmartin-tech
commented
6 months ago This looks to occur when an exception occurs and the backoff decorator intercepts, the generator instance is trying to be embedded in the exception being passed back to the primary process as the result object and is serialized via dumps().
A short term way to address this can be to clear the client on exception. Implementing a true custom pickle response could also mitigate this issue for the instance however it would not address other exceptions that might raise with object data that is not pickle safe either in dumps() or loads() of the object.
Once such instance is when a 4xx error occurs, notice this error is on the parent loads() side of the result processing:
Exception in thread Thread-4 (_handle_results):
Traceback (most recent call last):
File "/Users/jemartin/.pyenv/versions/3.10.14/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/Users/jemartin/.pyenv/versions/3.10.14/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/Users/jemartin/.pyenv/versions/3.10.14/lib/python3.10/multiprocessing/pool.py", line 579, in _handle_results
task = get()
File "/Users/jemartin/.pyenv/versions/3.10.14/lib/python3.10/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
TypeError: APIStatusError.__init__() missing 2 required keyword-only arguments: 'response' and 'body'More testing is in progress to determine if this if backoff is occurring in the parent or child process to determine how we might approach a general solution that can handle any exception that might occur in _call_model.
leondz
commented
6 months ago Interesting. Exception in _ForkingPickler.loads() seems telling. I can see some tests using request_mock becoming worthwhile to check behaviour of OpenAICompatible (and perhaps descendants) in response to 4xx, 5xx, other codes -- potentially under generations=1 and generations>1 conditions (do you think that part was relevant?)
jmartin-tech
commented
6 months ago Agreed, I will incorporate tests in my solution to add some mulitprocessing validation for each of the supported backoff exceptions and for HTTP error code responses.
OpenAIGenerator throws
"TypeError: cannot pickle '_thread.RLock' object"when invoked with--parallel_attempts> 1. The exception appears in a probe executed after a probe that involves only attempt.debugging journey follows:
at commit 5069362292fe2b674000cd25bd80621161515cf3, this test works (i.e. fails) consistently :
requesting just the single failing probe does no evoke the error:
really weird. it persists with
--parallel_attempts 2. it's also present for a dan probe, but again only seems to turn up if another probe has been run first?:putting the two "offending" probes one after another led to nothing:
but having one of the probes used before the failure, in first place, did summon an exception:
no idea if this is a red herring, but both probes called before the probes that exploded, have just one prompt (and this should skip the parallelisation that's invoked by
--parallel_attempts)trying again, it looks like it is a thing:
that's consistent enough for me - i'll re-write the summary at the top of this issue :)