TypeError: cannot pickle 'TLSBuffer' object

[elastictree]

Hello,

after starting the process (spawn) with an ff.Queue I get the following error. I am running on mac M1. Is fater-fifo in general compatible with the new Mac processors?

Thank you for your effort for the valuable project.

File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 284, in _Popen return Popen(process_obj) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__ super().__init__(process_obj) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__ self._launch(process_obj) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch reduction.dump(process_obj, fp) File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) TypeError: cannot pickle 'TLSBuffer' object

[alex-petrenko]

Hi @elastictree faster-fifo is known to work on M1 mac. Are you importing faster_fifo_reduction?

import faster_fifo_reduction

We define a custom pickler for the TLSBuffer ("Thread Local Storage Buffer"), it essentially just does not pickle it and just creates a new one when it's unpickled. Bottom line is you should never pickle TLSBuffers because threading.local is not pickleable.

[MosBas]

FWIW, I've seen the exact same issue with the M1 while using versions 1.4.1 and 1.4.2. I can confirm it works perfectly well with 1.4.0. Haven't had the chance to dig into the root cause, yet.

[alex-petrenko]

1.4.0 does not have a threading.local object, it was added to fix a bug that can cause issues when queues are used from multiple threads within one process. If you're only using faster-fifo with multiprocessing you should be fine with 1.4.0, otherwise, consider upgrading to the latest version.

Check if you're importing import faster_fifo_reduction to register a custom pickler for TLSBuffer. There are automatic tests for macOS that test this stuff and it works at least in the Github Actions environment. If this does not work, please post some code that reproduces the issue, I can look further into it! I'll have an access to M1 or M2 Mac in a couple of weeks.

[MosBas]

I think I was able to create a minimal repro, but it's not exactly the issue described above. Run this with 1.4.2:

from faster_fifo import Queue
import faster_fifo_reduction
import multiprocessing

class SubQueue(Queue):
    pass

def worker(x: Queue):
    pass

def main():
    q = SubQueue()  # Works with Queue()
    pool = multiprocessing.Pool(2, initializer=worker, initargs=(q,))

if __name__ == '__main__':
    main()

When sub-classing faster_fifo.Queue, you'll get the mentioned TypeError: cannot pickle 'TLSBuffer' object. Without it, it works fine. I assume this is because faster_fifo_reduction registers a custom pickler specifically for the Queue class (if I understand the source correctly). This can be overcome by repeating what it does manually.

[alex-petrenko]

@MosBas In hindsight, this is an obvious problem. Thank you for reporting! I went ahead and used a different way to work around TLSBuffer not being pickleable, by just defining __getstate__ and __setstate__ on it. I believe it is a much better solution than manual registration of the reduction function for the Queue object we used to do. It seems to work and now there's also no need to import faster_fifo_reduction.

Can you please download the latest code and see if it works for you? If there are no issues I will release a new PyPI version.

[MosBas]

Works like a charm :) Thanks!

[alex-petrenko]

https://pypi.org/project/faster-fifo/1.4.3/

Thanks again @MosBas