nusnlp / crosentgec

Code for cross-sentence grammatical error correction using multilayer convolutional seq2seq models (ACL 2019)
GNU General Public License v3.0
50 stars 12 forks source link

Multiprocessing error when attempting to run training on multiple GPUs #10

Closed salmatfq closed 3 years ago

salmatfq commented 3 years ago

Hi there,

When training the Cross Sentence GEC model on 1 GPU, it proceeds successfully. As soon as I try to use more than 1 GPU, I get the following error:

Traceback (most recent call last): File "/usr/lib/python3.6/pdb.py", line 1667, in main pdb._runscript(mainpyfile) File "/usr/lib/python3.6/pdb.py", line 1548, in _runscript self.run(statement) File "/usr/lib/python3.6/bdb.py", line 434, in run exec(cmd, globals, locals) File "", line 1, in File "/mnt/efs/crossSentGEC/fairseq/train.py", line 9, in import collections File "/mnt/efs/crossSentGEC/fairseq/multiprocessing_train.py", line 37, in main procs[i].start() File "/usr/lib/python3.6/multiprocessing/process.py", line 105, in start self._popen = self._Popen(self) File "/usr/lib/python3.6/multiprocessing/context.py", line 284, in _Popen return Popen(process_obj) File "/usr/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 32, in init super().init(process_obj) File "/usr/lib/python3.6/multiprocessing/popen_fork.py", line 19, in init self._launch(process_obj) File "/usr/lib/python3.6/multiprocessing/popen_spawn_posix.py", line 42, in _launch prep_data = spawn.get_preparation_data(process_obj._name) File "/usr/lib/python3.6/multiprocessing/spawn.py", line 172, in get_preparation_data main_mod_name = getattr(main_module.spec, "name", None) AttributeError: module 'main' has no attribute 'spec'

I am pretty sure that the underlying fairseq and Python multiprocessing modules are operational as I ran some sanity checks.

Any suggestions regarding how to solve/bypass this error please? And how I can run this with multi-GPUs?

Thank you all!