boto / s3transfer

Amazon S3 Transfer Manager for Python
Apache License 2.0
209 stars 133 forks source link

test_compat.py fails on MacOS and python 3.8+ #229

Open lamdor opened 2 years ago

lamdor commented 2 years ago

This is something that we started see in Nix when we enabled tests as part of the build: https://github.com/NixOS/nixpkgs/pull/154280

____ TestBaseManager.test_can_provide_signal_handler_initializers_to_start _____

self = <tests.unit.test_compat.TestBaseManager testMethod=test_can_provide_signal_handler_initializers_to_start>

    @skip_if_windows('os.kill() with SIGINT not supported on Windows')
    def test_can_provide_signal_handler_initializers_to_start(self):
        manager = self.create_pid_manager()
>       manager.start(signal.signal, (signal.SIGINT, signal.SIG_IGN))

tests/unit/test_compat.py:100:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/managers.py:554: in start
    self._process.start()
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/context.py:284: in _Popen
    return Popen(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <SpawnProcess name='PIDManager-43' parent=65299 initial>
file = <_io.BytesIO object at 0x1131d6630>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'TestBaseManager.create_pid_manager.<locals>.PIDManager'

/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/reduction.py:60: AttributeError

We think it might be due to spawn/fork changes in python 3.8, since it does succeed in python 3.7. https://github.com/NixOS/nixpkgs/pull/154280#issuecomment-1017180226 (ht to @thefloweringash for finding that)