Closed henrycoding closed 1 year ago
WIN下有相同问题 这样改只会bypass 【_mp_context = multiprocessing.get_context('spawn')】
但是如果真的使用multiprocess的话, 会出现异常
File "D:\hengzhang\code\ppio\venv\lib\site-packages\odps\models\instance.py", line 878, in to_pandas p.start() File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\multiprocessing\popen_spawn_win32.py", line 89, in init reduction.dump(process_obj, to_child) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'Instance._open_tunnel_reader..RecordReader.to_pandas..read_instance_split'
这是已知 bug,我们会在下个版本修复
Originally posted by @hekaisheng in https://github.com/aliyun/aliyun-odps-python-sdk/issues/165#issuecomment-946533521
文档里写了,目前是不支持 windows 多进程下载,这个还需要一些时间来支持
Fixed in v0.11.3.
WIN下有相同问题 这样改只会bypass 【_mp_context = multiprocessing.get_context('spawn')】
但是如果真的使用multiprocess的话, 会出现异常
File "D:\hengzhang\code\ppio\venv\lib\site-packages\odps\models\instance.py", line 878, in to_pandas p.start() File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\multiprocessing\process.py", line 112, in start self._popen = self._Popen(self) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\multiprocessing\popen_spawn_win32.py", line 89, in init reduction.dump(process_obj, to_child) File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'Instance._open_tunnel_reader..RecordReader.to_pandas..read_instance_split'
这是已知 bug,我们会在下个版本修复
Originally posted by @hekaisheng in https://github.com/aliyun/aliyun-odps-python-sdk/issues/165#issuecomment-946533521