GENRE + fairseq break in colab

[senisioi]

I've been trying to follow the setup described in the examples and to run a prediction example in colab.

Apparently, there are some issues installing the current branch in colab, here is a link to my notebook. Steps I tried:

1. Installing fairseq branch and running from fairseq import search gives a ModuleNotFoundError: No module named 'examples.simultaneous_translation'
2. I added the fairseq directory (with the examples dir) to the pythonpath and tried again
3. The setup_registry method returns None because the REGISTRY already exists. This breaks some call in fairseq/criterions/__init__.py

/usr/local/lib/python3.7/dist-packages/fairseq/criterions/__init__.py in <module>()
     22     CRITERION_DATACLASS_REGISTRY,
     23 ) = registry.setup_registry(
---> 24     "--criterion", base_class=FairseqCriterion, default="cross_entropy"
     25 )
     26 

TypeError: cannot unpack non-iterable NoneType object

[nicola-decao]

Have you used the indicated version of fairseq?

[nicola-decao]

Also the problem seems happening in from fairseq import search not in GENRE

[senisioi]

Yes, the right version of fairseq. I posted this issue here because the error appeared in relation to the specific branch of fairseq for this repository. The problem occurred when calling from fairseq_model import mGENRE

In the meantime, I got it to work, here's a colab example The problem was that in colab there is already an 'examples' library installed that is conflicting with this import: from examples.simultaneous_translation.utils.latency import LatencyTraining which triggered the first error ModuleNotFoundError: No module named 'examples.simultaneous_translation'. Removing the library before anything else solves the first issue.

To solve the error from the second step, I removed the two lines that return None when the criterion is already in the REGISTRY sed -i -e '26,27d' /content/fairseq/fairseq/registry.py

Thanks!

[dersuchendee]

It still doesn't work for me actually! See colab here

[senisioi]

The first line should be ! rm -rf /usr/local/lib/python3.10/dist-packages/examples to remove the pre-installed examples package.

[dersuchendee]

Thank you, but when adding pickles of wd and marisa_trie to try to get ids, it always runs out of RAM (even with colab Pro), how is that possible? Am I doing something wrong?

[senisioi]

Thank you, but when adding pickles of wd and marisa_trie to try to get ids, it always runs out of RAM (even with colab Pro), how is that possible? Am I doing something wrong?

Indeed, it's simply too large and unfortunately the whole thing is resource-prohibitive. I ended up renting a machine on vast.ai for a day