UKPLab / EasyNMT

Easy to use, state-of-the-art Neural Machine Translation for 100+ languages
Apache License 2.0
1.18k stars 118 forks source link

Offline installation #52

Open w3aryb0arpig opened 3 years ago

w3aryb0arpig commented 3 years ago

I am trying to install easyNMT on a semi-offline system. Python libraries are permitted to be installed but accessing other URLs is not permitted from this system. Therefore, can you advise on a way of how to manually install the Facebook (m2m_100_418M and m2m_100_1.2B) models so that easyNMT can see them? I can see they can be manually downloaded from Huggingface.co. If I were to download these on a second system, where should I then save them on the semi-offline Windows system on which I intend to test eastNMT? Thanks

nreimers commented 3 years ago

Hi @w3aryb0arpig

Download all the files from the hub. I think the following code should then work:

from easynmt import EasyNMT, models
model = EasyNMT(translator=models.AutoModel("path/to/model"))
w3aryb0arpig commented 3 years ago

@nreimers this works a treat but I hit another issue with fasttext being called by the script:-

    print("=> detected language:", MODEL.language_detection(sentence), "\n")
  File "C:\Program Files\Python38\lib\site-packages\easynmt\EasyNMT.py", line 413, in language_detection
    raise Exception("No method for automatic language detection was found. Please install at least one of the following: fasttext (pip install fasttext), langid (pip install langid), or langdetect (pip install langdetect)")
Exception: No method for automatic language detection was found. Please install at least one of the following: fasttext (pip install fasttext), langid (pip install langid), or langdetect (pip install langdetect)

I actually have separate code in my script using fasttext successfully and it is installed. I think the issue lies in the fact that as this is an offline system, the _lid.176.ftz__ file isn't in the correct path neither for where the easynmt.py script is looking for it.

Looking at the easyNMT.py, the relevant segment appears to be:-


        if self._fasttext_lang_id is None:
            import fasttext
            fasttext.FastText.eprint = lambda x: None   #Silence useless warning: https://github.com/facebookresearch/fastText/issues/1067
            model_path = os.path.join(self._cache_folder, 'lid.176.ftz')

So it would appear that _cache_folder is where the file should be stored. I can't work out where this _cache_folder is though? I did find the below but couldn't work out where this related to:-


        if cache_folder is None:
            if 'EASYNMT_CACHE' in os.environ:
                cache_folder = os.environ['EASYNMT_CACHE']
            else:
                cache_folder = os.path.join(torch.hub._get_torch_home(), 'easynmt_v2')
        self._cache_folder = cache_folder

Any ideas @nreimers where I can store the lid.176.ftz on a Windows system manually?

nreimers commented 3 years ago

You can set the environment variable EASYNMT_CACHE to point to any folder you like.

Otherwise, when you call translate and pass the source_lang, then no automatic language detection happens. So you could wrap your translate function, that you do the language detection (if needed) and pass it to model.translate

w3aryb0arpig commented 3 years ago

Further to the above, I managed to address this by using the cache_folder parameter:-

MODEL = EasyNMT(translator=models.AutoModel("models//facebook//m2m100_418M"), cache_folder='cached')

Tortoise17 commented 3 years ago

Is there the way to download all models one time from hub by any CLI? to specific path= ?

nreimers commented 3 years ago

@Tortoise17 You can use standard git checkout to get the model files:

git clone https://huggingface.co/facebook/m2m100_1.2B

will download all files from https://huggingface.co/facebook/m2m100_1.2B

Tortoise17 commented 3 years ago

Thank you. I hope that it has no limitations in terms of usage of text (except the size of the text chunk per input cli).

nreimers commented 3 years ago

Sadly I don't understand what you mean

Tortoise17 commented 3 years ago

Is there any limit in the use of the model for translation when the transformer is used with the use of internet?? Limit means the characters per month or per day or CLI use per day limit?

nreimers commented 3 years ago

No, there is no such limit

Tortoise17 commented 3 years ago

@Tortoise17 You can use standard git checkout to get the model files:

git clone https://huggingface.co/facebook/m2m100_1.2B

will download all files from https://huggingface.co/facebook/m2m100_1.2B

Thank you. and how to get all Opus-mt models download by CLI?

nreimers commented 3 years ago

You have to download each individually

Tortoise17 commented 3 years ago

I am trying to keep the model downloaded in the local disk as I have no space in native cache. I am trying like this, can you guide how this model I can save locally and than load from locally ?

I cloned and git-lfs installed. than

from easynmt import EasyNMT, models
model = EasyNMT(translator=models.AutoModel("/home/translate/m2m100_418M")), cache_folder='/home/translate/cached')

First it demaded the easynmt.json in m2m100_418M folder.

If you can guide once for opus-mt and m2m100_418M model. I assume that the way should be same for both.

I am confused where is the mistake.

ArchanGhosh commented 2 years ago

@nreimers this works a treat but I hit another issue with fasttext being called by the script:-

    print("=> detected language:", MODEL.language_detection(sentence), "\n")
  File "C:\Program Files\Python38\lib\site-packages\easynmt\EasyNMT.py", line 413, in language_detection
    raise Exception("No method for automatic language detection was found. Please install at least one of the following: fasttext (pip install fasttext), langid (pip install langid), or langdetect (pip install langdetect)")
Exception: No method for automatic language detection was found. Please install at least one of the following: fasttext (pip install fasttext), langid (pip install langid), or langdetect (pip install langdetect)

I actually have separate code in my script using fasttext successfully and it is installed. I think the issue lies in the fact that as this is an offline system, the _lid.176.ftz__ file isn't in the correct path neither for where the easynmt.py script is looking for it.

Looking at the easyNMT.py, the relevant segment appears to be:-


        if self._fasttext_lang_id is None:
            import fasttext
            fasttext.FastText.eprint = lambda x: None   #Silence useless warning: https://github.com/facebookresearch/fastText/issues/1067
            model_path = os.path.join(self._cache_folder, 'lid.176.ftz')

So it would appear that _cache_folder is where the file should be stored. I can't work out where this _cache_folder is though? I did find the below but couldn't work out where this related to:-


        if cache_folder is None:
            if 'EASYNMT_CACHE' in os.environ:
                cache_folder = os.environ['EASYNMT_CACHE']
            else:
                cache_folder = os.path.join(torch.hub._get_torch_home(), 'easynmt_v2')
        self._cache_folder = cache_folder

Any ideas @nreimers where I can store the lid.176.ftz on a Windows system manually?

Hi, I was getting the same issue, you can store the ftz file in the same cache folder, after which there would be two outcomes,

  1. Your model is working
  2. You are getting the same error

for me it was completely random so what I did was, model_path = os.path.join(self._cache_folder, 'lid.176.ftz'), I harded the path, which is obviously not the right way but I also had a restricted machine and I couldn't find a work-around with my limited knowledge.

Hope this helps