Since make_sentinel uses a local class defined within the function body, sentinels are not picklable, and as a consequence, glom can't easily be used with muliprocessing:
Traceback (most recent call last):
File "bin/testclient.py", line 200, in <module>
main()
File "bin/testclient.py", line 123, in main
pool.map(task.run, get_mongo_batches()),
File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
File "/usr/lib/python3.8/multiprocessing/pool.py", line 537, in _handle_tasks
put(task)
File "/usr/lib/python3.8/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'make_sentinel.<locals>.Sentinel'
Oh, nevermind, this is actually an issue with boltons. The point remains that it makes glom impossible to use with multiprocessing, but the root cause is elsewhere.
Since
make_sentinel
uses a local class defined within the function body, sentinels are not picklable, and as a consequence,glom
can't easily be used withmuliprocessing
: