Closed Oaklight closed 2 years ago
after some digging, it's very likely caused by the jupyter/ipython's handling of spawn:
Similar situations got discussed in several places. https://github.com/pytorch/pytorch/issues/17680#issuecomment-491384202 for example.
I'm testing the above code in .py
script, and will try to modify parallelformer
's parallelize.py
to replace torch.multiprocessing
with multiprocess
unfortunately, directly running the above code in script mode throw this error
$ python megatron-11b-prompt.py
Traceback (most recent call last):
File "/homes/pding/miniconda3/envs/megatron/lib/python3.8/site-packages/parallelformers/parallelize.py", line 283, in parallelize
process.start()
File "/homes/pding/miniconda3/envs/megatron/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/homes/pding/miniconda3/envs/megatron/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/homes/pding/miniconda3/envs/megatron/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/homes/pding/miniconda3/envs/megatron/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/homes/pding/miniconda3/envs/megatron/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/homes/pding/miniconda3/envs/megatron/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
File "/homes/pding/miniconda3/envs/megatron/lib/python3.8/multiprocessing/spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "/homes/pding/miniconda3/envs/megatron/lib/python3.8/multiprocessing/spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
ok, fixed by replacing torch.multiprocessing
with multiprocess
in parallelize.py
PR open at #21
Didn't these changes cause any other problems?
I didn't see other error
Ok I see. thank you.
Trying to use parallelformers with the megatron-11b pip package. The MegatronPolicy class is as-provided from megatron-11b pypi webpage
How to reproduce
Environment