ModuleNotFoundError: No module named 'examples.simultaneous_translation'

[RSid8]

🐛 Bug

Performing transfer learning using Roberta by following the custom classification readme in the Examples directory of Roberta. This code was working up to 1 week ago and now gives an error: ModuleNotFoundError: No module named 'examples.simultaneous_translation'

Steps to reproduce the behavior:

fairseq-preprocess \
    --only-source \
    --trainpref "/content/Task1a/train.input0.bpe" \
    --validpref "/content/Task1a/val.input0.bpe" \
    --destdir "/content/Task1a-bin/input0" \
    --workers 60 \
    --srcdict dict.txt

fairseq-preprocess \
    --only-source \
    --trainpref "/content/Task1a/train.label" \
    --validpref "/content/Task1a/val.label" \
    --destdir "/content/Task1a-bin/label" \
    --workers 60

2. Traceback:

   
   --2021-02-28 05:54:18--  https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/dict.txt
   Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 104.22.75.142, 172.67.9.4, 104.22.74.142, ...
   Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|104.22.75.142|:443... connected.
   HTTP request sent, awaiting response... 304 Not Modified
   File ‘dict.txt’ not modified on server. Omitting download.

Traceback (most recent call last): File "/usr/local/bin/fairseq-preprocess", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "/usr/local/bin/fairseq-preprocess", line 25, in importlib_load_entry_point return next(matches).load() File "/usr/local/lib/python3.7/dist-packages/importlib_metadata/init.py", line 100, in load module = import_module(match.group('module')) File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "", line 728, in exec_module File "", line 219, in _call_with_frames_removed File "/content/fairseq/fairseq_cli/preprocess.py", line 18, in from fairseq import options, tasks, utils File "/content/fairseq/fairseq/init.py", line 32, in import fairseq.criterions # noqa File "/content/fairseq/fairseq/criterions/init.py", line 36, in importlib.import_module("fairseq.criterions." + file_name) File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/content/fairseq/fairseq/criterions/label_smoothed_cross_entropy_latency_augmented.py", line 6, in from examples.simultaneous_translation.utils.latency import LatencyTraining ModuleNotFoundError: No module named 'examples.simultaneous_translation' Traceback (most recent call last): File "/usr/local/bin/fairseq-preprocess", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "/usr/local/bin/fairseq-preprocess", line 25, in importlib_load_entry_point return next(matches).load() File "/usr/local/lib/python3.7/dist-packages/importlib_metadata/init.py", line 100, in load module = import_module(match.group('module')) File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1006, in _gcd_import File "", line 983, in _find_and_load File "", line 967, in _find_and_load_unlocked File "", line 677, in _load_unlocked File "", line 728, in exec_module File "", line 219, in _call_with_frames_removed File "/content/fairseq/fairseq_cli/preprocess.py", line 18, in from fairseq import options, tasks, utils File "/content/fairseq/fairseq/init.py", line 32, in import fairseq.criterions # noqa File "/content/fairseq/fairseq/criterions/init.py", line 36, in importlib.import_module("fairseq.criterions." + file_name) File "/usr/lib/python3.7/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/content/fairseq/fairseq/criterions/label_smoothed_cross_entropy_latency_augmented.py", line 6, in from examples.simultaneous_translation.utils.latency import LatencyTraining ModuleNotFoundError: No module named 'examples.simultaneous_translation'

### Expected behavior

Expected to save the preprocessed and tokenized input and labels for training

### Environment

 - fairseq Version (master):
 - Google Colab

[lematt1991]

Can you try re-running pip install . --editable. I'm unable to reproduce this on my side.

[KirillKonovalov]

I had the same issue. What can be done?

[RSid8]

@lematt1991 I did try that but it failed to work. However, I did a regular

!pip install fairseq

and cloned the repository after. This caused the error to stop appearing. @KirillKonovalov that might be helpful to you.

[benjamin3344]

@lematt1991 I did try that but it failed to work. However, I did a regular

!pip install fairseq

and cloned the repository after. This caused the error to stop appearing. @KirillKonovalov that might be helpful to you.

If I am installing and running fairseq locally (installing from setup.py), how can I solve this problem? @RSid8

[wesleywt]

Adding fairseq to the PYTHONPATH solved the problem for me. I got the solution from this issue

cd /content 
!echo $PYTHONPATH

import os

os.environ['PYTHONPATH'] += ":/content/fairseq/" #path to your fairseq 

!echo $PYTHONPATH

[aanchan]

This probably is not a bug. The examples folder falls outside of the purview of what is "included" in the package. If you check at the repo level its fairseq/fairseq and fairseq/examples. Anything inside fairseq/fairseq appears importable. So fairseq/examples will not be importable.

[wangtianrui]

A simple solution inspired by @wesleywt :

PYTHONPATH=/home/wang/codes/py/fairseq python -u examples/speech_to_speech/preprocessing/prep_s2ut_data.py 

[donglixp]

FAIRSEQ_DIR=$(pip list -v | grep 'fairseq' | awk '{print $3}')
export PYTHONPATH=$PYTHONPATH:$FAIRSEQ_DIR