Error when training WMT with multi gpu

[isanvicente]

Hi!

my setup:

• ubuntu 22.04
• Python 3.10
• Torch 2.3.1

• eole 0.0.2

  I was trying to follow the wmt17 en-de recipe, in multi-gpu (2 gpus - world size: 2; gpu_ranks:[0,1]):, but I got the following error when running eole train command:

Traceback (most recent call last):
  File "/mnt/nfs/NMT/eole/eole/utils/distributed_workers.py", line 48, in spawned_train
    process_fn(config, device_id=device_id)
  File "/mnt/nfs/NMT/eole/eole/train_single.py", line 242, in main
    trainer.train(
  File "/mnt/nfs/NMT/eole/eole/trainer.py", line 337, in train
    eole.utils.distributed.all_gather_list(normalization)
AttributeError: module 'eole.utils' has no attribute 'distributed'

    self._shutdown_workers()
  File "/mnt/nfs/NMT/venv_eole_gpu4/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1443, in _shutdown_workers
    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
  File "/usr/lib/python3.10/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
  File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 40, in wait
    if not wait([self.sentinel], timeout):
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.10/selectors.py", line 416, in select
    fd_event_list = self._selector.poll(timeout)
KeyboardInterrupt: 
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)

solved it adding the import (from eole.utils import distributed) to trainer.py

Hope it helps,

cheers!

[francoishernandez]

Thanks for the report! It seems this issue was introduced in #116 which removed some explicit imports that hid the wonky behaviour. Would you mind opening a quick PR with your fix? Thanks!

[isanvicente]

Will do! thanks for the quick response!