mahmoud / glom

☄️ Python's nested data operator (and CLI), for all your declarative restructuring needs. Got data? Glom it! ☄️
https://glom.readthedocs.io
Other
1.89k stars 61 forks source link

Sentinels are not picklable #222

Closed mathrick closed 3 years ago

mathrick commented 3 years ago

Since make_sentinel uses a local class defined within the function body, sentinels are not picklable, and as a consequence, glom can't easily be used with muliprocessing:

Traceback (most recent call last):
  File "bin/testclient.py", line 200, in <module>
    main()
  File "bin/testclient.py", line 123, in main
    pool.map(task.run, get_mongo_batches()),
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 537, in _handle_tasks
    put(task)
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'make_sentinel.<locals>.Sentinel'
mathrick commented 3 years ago

Oh, nevermind, this is actually an issue with boltons. The point remains that it makes glom impossible to use with multiprocessing, but the root cause is elsewhere.

mathrick commented 3 years ago

Created mahmoud/boltons#285 instead.