shutil.Error: Destination path '/home/ubuntu/.cache/huggingface/transformers/transformers' already exists

[parthplc]

Environment info

• transformers version:
• Platform: Ubuntu 18.04
• Python version: 1.7.0
• PyTorch version (GPU?): 1.7.0
• Tensorflow version (GPU?):
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help

@patrickvonplaten @sgugger -->

Information

Model I am using (MT5ForConditionalGeneration.):

The problem arises when using: I am trying to run my script importing Mt5

In Transformers v4.0.0, the default path to cache downloaded models changed from '~/.cache/torch/transformers' to '~/.cache/huggingface/transformers'. Since you don't seem to have overridden and '~/.cache/torch/transformers' is a directory that exists, we're moving it to '~/.cache/huggingface/transformers' to avoid redownloading models you have already in the cache. You should only see this message once.
Traceback (most recent call last):
  File "__main__.py", line 87, in <module>
    from data_science.recommenders.content_recommender.context_similarity import Context_Similarity
  File "/home/ubuntu/parth/trell-ds-framework/data_science/recommenders/content_recommender/context_similarity.py", line 5, in <module>
    from sentence_transformers import SentenceTransformer
  File "/home/ubuntu/venv_trellai/lib/python3.6/site-packages/sentence_transformers/__init__.py", line 3, in <module>
    from .datasets import SentencesDataset, SentenceLabelDataset
  File "/home/ubuntu/venv_trellai/lib/python3.6/site-packages/sentence_transformers/datasets.py", line 12, in <module>
    from . import SentenceTransformer
  File "/home/ubuntu/venv_trellai/lib/python3.6/site-packages/sentence_transformers/SentenceTransformer.py", line 10, in <module>
    import transformers
  File "/home/ubuntu/venv_trellai/lib/python3.6/site-packages/transformers/__init__.py", line 22, in <module>
    from .integrations import (  # isort:skip
  File "/home/ubuntu/venv_trellai/lib/python3.6/site-packages/transformers/integrations.py", line 5, in <module>
    from .trainer_utils import EvaluationStrategy
  File "/home/ubuntu/venv_trellai/lib/python3.6/site-packages/transformers/trainer_utils.py", line 25, in <module>
    from .file_utils import is_tf_available, is_torch_available, is_torch_tpu_available
  File "/home/ubuntu/venv_trellai/lib/python3.6/site-packages/transformers/file_utils.py", line 227, in <module>
    shutil.move(old_default_cache_path, default_cache_path)
  File "/usr/lib/python3.6/shutil.py", line 548, in move
    raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path '/home/ubuntu/.cache/huggingface/transformers/transformers' already exists

To reproduce

I am using Transformer==4.0.0 I get this error but when installing transformers==4.0.0rc1 the error doesn't show. Is there any reason for this?

[sgugger]

This was a bug added in v4, it's fixed on master so if you install from source, you should be fine.

[patrickvonplaten]

I think alternatively you could also just delete the cache:

rm -rf /home/ubuntu/.cache/huggingface/transformers/transformers

but then you'll have to re-download all models

[parthplc]

yeah its worked!

[LysandreJik]

@patrickvonplaten since this means that the cache has already been moved to .cache/huggingface/transformers, I think deleting the cache .cache/torch/transformers makes more sense, as you won't have to delete all the models you had in the initial cache, only those that were redownloaded when you went back to an older version.

[ZihaoZheng98]

Why do transformers throw an error ? If it exists,why not just throw a warning?I think this is a bug and should be fixed.

[sgugger]

Please read the full conversation:

This was a bug added in v4, it's fixed on master so if you install from source, you should be fine.