clip adaptor

[haoxiangzhao12138]

In the Clip Adapter, which specific model of OpenCLIP is used? When using torch.hub.load with adaptor_names='clip', it requires access to Hugging Face. Is there a way to load the Clip Adapter locally instead?

[mranzinger]

Hmm, are you sure it requires huggingface? It should be using open_clip: https://github.com/NVlabs/RADIO/blob/354971da4b39b66c4daf79c4e48394ba2e28cbc4/radio/open_clip_adaptor.py#L23-L29

And this is the specific model we use: https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_results.csv#L2

name and pretrained map to model_name and pretrained here: https://github.com/NVlabs/RADIO/blob/354971da4b39b66c4daf79c4e48394ba2e28cbc4/radio/open_clip_adaptor.py#L26-L27

[haoxiangzhao12138]

Thank you very much for your response. When I use torch.hub.load(... , ... , source='local') to load a local model and add the adaptor_name parameter, I encounter the following error:

requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /apple/DFN5B-CLIP-ViT-H-14-378/resolve/main/open_clip_pytorch_model.bin (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f422eedcb50>, 'Connection to huggingface.co timed out. (connect timeout=10)'))"), '(Request ID: cf1f5d30-1fe2-4417-abde-ed366c875d0a)')

From what I understand, it seems like there’s an issue accessing Hugging Face. Is there any additional parameter I might need to add to torch.hub.load to address this?

[mranzinger]

Ah, I see. So open_clip is pulling the model weights from huggingface. Do you have access to any environment that is able to access them? Do you need to text encoder for the clip model?