When training the Cross Sentence GEC model on 1 GPU, it proceeds successfully. As soon as I try to use more than 1 GPU, I get the following error:
Traceback (most recent call last):
File "/usr/lib/python3.6/pdb.py", line 1667, in main
pdb._runscript(mainpyfile)
File "/usr/lib/python3.6/pdb.py", line 1548, in _runscript
self.run(statement)
File "/usr/lib/python3.6/bdb.py", line 434, in run
exec(cmd, globals, locals)
File "", line 1, in
File "/mnt/efs/crossSentGEC/fairseq/train.py", line 9, in
import collections
File "/mnt/efs/crossSentGEC/fairseq/multiprocessing_train.py", line 37, in main
procs[i].start()
File "/usr/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/usr/lib/python3.6/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/usr/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/usr/lib/python3.6/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/usr/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
File "/usr/lib/python3.6/multiprocessing/spawn.py", line 172, in get_preparation_data
main_mod_name = getattr(main_module.spec, "name", None)
AttributeError: module 'main' has no attribute 'spec'
I am pretty sure that the underlying fairseq and Python multiprocessing modules are operational as I ran some sanity checks.
Any suggestions regarding how to solve/bypass this error please? And how I can run this with multi-GPUs?
Hi there,
When training the Cross Sentence GEC model on 1 GPU, it proceeds successfully. As soon as I try to use more than 1 GPU, I get the following error:
I am pretty sure that the underlying fairseq and Python multiprocessing modules are operational as I ran some sanity checks.
Any suggestions regarding how to solve/bypass this error please? And how I can run this with multi-GPUs?
Thank you all!