Can't download R3M models

dtch1997 commented 9 months ago

To reproduce:

$ python src/jam/scripts/download_and_convert.py --model r3m/r3m-18

Error message:

Cached Downloading: data/models/r3m/r3m-18/torch_model.pt                                                                                                                                 
Access denied with the following error:                                                                                                                                                   

        Cannot retrieve the public link of the file. You may need to change                                                                                                               
        the permission to 'Anyone with the link', or have had many accesses. 

You may still be able to access the file from the browser:

         https://drive.google.com/uc?id=1A1ic-p4KtYlKXdXHcV2QV0cUzI4kn0u-

Stack trace:

Traceback (most recent call last):
  File "/usr/lib/python3.10/shutil.py", line 816, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: '/home/daniel/.cache/gdown/tmpjth_2qk3/dl' -> 'data/models/r3m/r3m-18/torch_model.pt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/daniel/jam/src/jam/scripts/download_and_convert.py", line 26, in <module>
    app.run(main)
  File "/home/daniel/jam/venv/lib/python3.10/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/daniel/jam/venv/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/home/daniel/jam/src/jam/scripts/download_and_convert.py", line 22, in main
    model_zoo.download_and_convert(model_name, _MODEL_DIR.value)
  File "/home/daniel/jam/src/jam/model_zoo.py", line 140, in download_and_convert
    gdown.cached_download(
  File "/home/daniel/jam/venv/lib/python3.10/site-packages/gdown/cached_download.py", line 117, in cached_download
    shutil.move(temp_path, path)
  File "/usr/lib/python3.10/shutil.py", line 836, in move
    copy_function(src, real_dst)
  File "/usr/lib/python3.10/shutil.py", line 434, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.10/shutil.py", line 254, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/home/daniel/.cache/gdown/tmpjth_2qk3/dl'

dtch1997 commented 9 months ago

I strongly suspect this is something to do with the R3M authors' google drive link, not gdown, because all the other model downloads work.

EDIT: I can still download the model.pt manually by opening the download link in a web browser. I guess the download is rate-limited or something.

dtch1997 commented 9 months ago

I tried to circumvent this issue by creating my own copy of the file.

Downloaded the r3m/r3m-18 model from the original link.
Uploaded it to my own Google Drive here
Created a new direct download link using this utility

In jam.model_zoo, updated the URL.

"r3m/r3m-18": {
    # Original URL from R3M
    # "ckpt_url": "https://drive.google.com/uc?id=1A1ic-p4KtYlKXdXHcV2QV0cUzI4kn0u-",
    "ckpt_url": "https://drive.google.com/uc?export=download&id=1FhMe4QOzLs0OVRww9mNBK54T6R6xiALt",
    "ckpt_md5": "777854e2548f91480b0bd538a06ba017",
    # "config_url": "https://drive.google.com/uc?id=1nitbHQ-GRorxc7vMUiEHjHWP5N11Jvc6",
    # "config_md5": "e226d5115543dee07fd51a17c7866bc9",
},

I still get the same error message. Very odd...

dtch1997 commented 9 months ago

A possible solution might be to manually download the file, then convert it according to jam. This would require disentangling the download logic from the convert logic.

ethanluoyc commented 9 months ago

I have been thinking about hosting a converted copy of the model weights (in safetensor or npz) on HF, which would streamline loading quite a bit, what are your thoughts?

dtch1997 commented 9 months ago

Sorry, missed this earlier! Yeah I think hosting a copy of the weights is a great idea, assuming there are no licensing issues

ethanluoyc / jam

Can't download R3M models #3