search

GoogleCloudPlatform / gsutil

A command line tool for interacting with cloud storage services.
Apache License 2.0
875 stars 336 forks source link

gsutil flag '-m' exception with Python 3.8 #1085

Open ghost opened 4 years ago

ghost commented 4 years ago

gsutil works with Python 8 until you use the '-m' flag:

python -V -> Python 3.8.3

Your current Cloud SDK version is: 304.0.0

gsutil -m cp -r gs://foo-bar/* .
Traceback (most recent call last):
  File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gsutil", line 21, in <module>
    gsutil.RunMain()
  File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gsutil.py", line 123, in RunMain
    sys.exit(gslib.__main__.main())
  File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 429, in main
    return _RunNamedCommandAndHandleExceptions(
  File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 767, in _RunNamedCommandAndHandleExceptions
    _HandleUnknownFailure(e)
  File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/__main__.py", line 625, in _RunNamedCommandAndHandleExceptions
    return command_runner.RunNamedCommand(command_name,
  File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command_runner.py", line 411, in RunNamedCommand
    return_code = command_inst.RunCommand()
  File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/commands/cp.py", line 1190, in RunCommand
    self.Apply(_CopyFuncWrapper,
  File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1499, in Apply
    self._ParallelApply(
  File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1719, in _ParallelApply
    self._CreateNewConsumerPool(process_count, thread_count,
  File "/Users/mi/tools/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1384, in _CreateNewConsumerPool
    p.start()
  File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/mi/.pyenv/versions/3.8.3/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle '_io.TextIOWrapper' object
turribeach commented 4 years ago

Probably better if you post the same command with -d added to get debug info.

ghost commented 4 years ago

gsutil -d -m cp -r gs://foo-bar/ . gsutil -d cp -r gs://foo-bar/ .

Without '-m' works fine, see attached log for with '-m'

debug.log

djc commented 4 years ago

This looks like a duplicate of https://github.com/GoogleCloudPlatform/gsutil/issues/961.