Pickling error when using background_callback with process pool executor

[TylerADavis]

Background callbacks work fine and process pool executor works fine, but when used together I get the following error. I have now run into this issue both on macOS (python 3.6) and ubuntu (python 3.5).

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/queues.py", line 234, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'FuturesSession.request.<locals>.wrap'

Below is a reproducible example:

from concurrent.futures import ProcessPoolExecutor
from requests_futures.sessions import FuturesSession
from requests import Session

def bg_cb(sess, resp):
    print('background callback successful')

session = FuturesSession(executor=ProcessPoolExecutor(max_workers=2),
                         session=Session())

future = session.get('https://www.google.com', background_callback=bg_cb)

print(future.result())

[ross]

AttributeError: Can't pickle local object 'FuturesSession.request..wrap'

Hrm. That looks like it's failing to pickle the wrapper function. I don't imagine it's possible to pickle functions, but can't say that it's something I've directly looked into.

You might look into the built in requests callback stuff, http://docs.python-requests.org/en/master/user/advanced/#event-hooks. To be honest if I had it to do over I wouldn't add the background_callback stuff here as I think those hooks would be a better way to do most of what you would do in them anyway.

[TylerADavis]

Great, thank you for the help! Using the built in hooks system I was able to get what I’d wanted to done (just background parsing of json). The one catch is I ran into issues with actually passing the information back out, I believe due to how the response is pickled and moved between processes. I ended up saying response.elapsed = response.json(). I chose this field as it was actually modifiable, and the other fields were not. Trying to add a new field to the response resulted in the data being lost when using a processpool, although it worked fine with a threadpool.

[ross]

Trying to add a new field to the response resulted in the data being lost when using a processpool, although it worked fine with a threadpool.

Funky & good to know. I think i'm going to go ahead and deprecate background_callback itself and point people towards the callbacks. Sounds like they're workable with processes and they should be a drop-in replacement and better option for threadpools.

[FJunior225]

Hello! I have the latest version and getting a similar error when trying to use your advanced example for the hooks - AttributeError: Can't pickle local object 'ElapsedFuturesSession.request..timing'

class ElapsedFuturesSession(FuturesSession):

    def request(self, method, url, hooks={}, *args, **kwargs):
        start = time()

        def timing(r, *args, **kwargs):
            r.elapsed = time() - start

        try:
            if isinstance(hooks['response'], (list, tuple)):
                # needs to be first so we don't time other hooks execution
                hooks['response'].prepend(timing)
            else:
                hooks['response'] = [timing, hooks['response']]
        except KeyError:
            hooks['response'] = timing

        return super(ElapsedFuturesSession, self) \
            .request(method, url, hooks=hooks, *args, **kwargs)

My main goal here is to track the time it takes to send a request to a downstream. I also seen another attribute error, but I believe the above issue is the main concern. hooks['response'].prepend(timing) AttributeError: 'list' object has no attribute 'prepend'

Thoughts? Or suggestions?

Thanks!

[ross]

AttributeError: 'list' object has no attribute 'prepend'

Too many languages with slight diffs and not enough example testing. Looks like I didn't hit that case of the code. That line should be

                    hooks['response'].insert(0, timing)

Will get the readme updated. Thanks.

[FJunior225]

Thank you!

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.