search

neulab / awesome-align

A neural word aligner based on multilingual BERT
https://arxiv.org/abs/2101.08231
BSD 3-Clause "New" or "Revised" License
325 stars 47 forks source link

AttributeError: Can't pickle local object 'word_align.<locals>.collate' #28

Closed manandey closed 3 years ago

manandey commented 3 years ago

Hi,

When I try to extract the alignments using the command given in the readme here, I get the following error:

Loading the dataset...
Extracting: 0it [00:00, ?it/s]Traceback (most recent call last):
  File "C:\Users\I355109\Anaconda3\Scripts\awesome-align-script.py", line 33, in <module>
    sys.exit(load_entry_point('awesome-align==0.1.6', 'console_scripts', 'awesome-align')())
  File "C:\Users\I355109\Anaconda3\lib\site-packages\awesome_align-0.1.6-py3.8.egg\awesome_align\run_align.py", line 294, in main
  File "C:\Users\I355109\Anaconda3\lib\site-packages\awesome_align-0.1.6-py3.8.egg\awesome_align\run_align.py", line 171, in word_align
  File "C:\Users\I355109\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 352, in __iter__
    return self._get_iterator()
  File "C:\Users\I355109\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 294, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\I355109\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 801, in __init__
    w.start()
  File "C:\Users\I355109\Anaconda3\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\I355109\Anaconda3\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\I355109\Anaconda3\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\I355109\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\I355109\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'word_align.<locals>.collate'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\I355109\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\I355109\Anaconda3\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
Extracting: 0it [00:01, ?it/s]

Can you please help me figure out if I am doing anything wrong?

zdou0830 commented 3 years ago

Hi, how many lines are there in your file? could you add --num_workers 1 to your command and try again? thanks!

manandey commented 3 years ago

I tried adding --num_workers 1 to my command but I am facing the same issue. There are around 3168 lines in my file.

zdou0830 commented 3 years ago

Hi, just to confirm, will you face this issue if you align the files in the examples folder (e.g. examples/enfr.src-tgt) and would add --num_workers 0 to your command work? If still not, could you share more details of your environment (e.g.your python and pytorch version)? thanks!

manandey commented 3 years ago

It worked! Thanks a lot @zdou0830!