Host models on HF - Githubissues

NielsRogge commented 1 week ago

Hi @bhheo,

Niels here from the open-source team at Hugging Face. I found your work through ECCV (congrats!), and indexed your paper here: https://huggingface.co/papers/2403.13298, congrats on getting it accepted to ECCV (feel free to claim authorship with your HF account). I work together with AK on improving the visibility of researchers' work on the hub.

I was wondering you'd be up for collaborating on releasing the checkpoints on the 🤗 hub, rather than Google Drive, to improve their discoverability. We can add tags so that people find them when filtering https://huggingface.co/models.

Uploading models

See here for a guide: https://huggingface.co/docs/hub/models-uploading. In case the models are custom PyTorch model, we could probably leverage the PyTorchModelHubMixin class which adds from_pretrained and push_to_hub to each model. Alternatively, one can leverages the hf_hub_download one-liner to download a checkpoint from the hub.

We encourage researchers to push each model checkpoint to a separate model repository, so that things like download stats also work. Moreover, we can then link the checkpoints to the paper page, improving their visibility.

Let me know if you're interested/need any help regarding this!

Cheers,

Niels ML Engineer @ HF 🤗

bhheo commented 6 days ago

Hi Niels

Thank you for good suggestion I totally agree that hf is better than google drive link I will try it with PyTorchModelHubMixin and ask you if I meet any problem

Best Heo

NielsRogge commented 5 days ago

Thank you! Btw would be great to push them to https://huggingface.co/naver

bhheo commented 4 days ago

Hi @NielsRogge

I have an issue while trying to convert my model to safetensors. Here's the code I'm using:

import huggingface_hub
huggingface_hub.login("")

import torch
from models import vit_rope

model = vit_rope.rope_axial_deit_small_patch16_LS()

# load checkpoint -> push to hub
checkpoint = torch.load("/mnt/tmp/weights/rope-vit/rope_axial_deit_small_patch16_LS.pth")
model.load_state_dict(checkpoint['model'])
model.save_pretrained("rope_axial_deit_small_patch16_LS")
# model.push_to_hub("bhheo/rope_axial_deit_small_patch16_LS")

# hub to model
model.from_pretrained("rope_axial_deit_small_patch16_LS")

But, I keep getting following error at the last line. Could you advise on what might be causing this and how to fix it?

  File "/mnt/image-net-full/bhheo/rope-vit/test.py", line 16, in <module>
    model.from_pretrained("rope_axial_deit_small_patch16_LS")
  File "/home/nsml/.local/lib/python3.8/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/nsml/.local/lib/python3.8/site-packages/huggingface_hub/hub_mixin.py", line 570, in from_pretrained
    instance = cls._from_pretrained(
  File "/home/nsml/.local/lib/python3.8/site-packages/huggingface_hub/hub_mixin.py", line 794, in _from_pretrained
    return cls._load_as_safetensor(model, model_file, map_location, strict)
  File "/home/nsml/.local/lib/python3.8/site-packages/huggingface_hub/hub_mixin.py", line 843, in _load_as_safetensor
    safetensors.torch.load_model(model, model_file, strict=strict, device=map_location)  # type: ignore [arg-type]
  File "/home/nsml/.local/lib/python3.8/site-packages/safetensors/torch.py", line 202, in load_model
    state_dict = load_file(filename, device=device)
  File "/home/nsml/.local/lib/python3.8/site-packages/safetensors/torch.py", line 315, in load_file
    result[k] = f.get_tensor(k)
RuntimeError: Viewing a tensor as a new dtype with a different number of bytes per element is not supported.

NielsRogge commented 4 days ago

Hi,

this is probably because of the fact that the model takes various arguments in its init method which aren't JSON serializable (like classes which are of type nn.Module). Hence it cannot serialize those to a config.json.

One could probably add a custom from_pretrained method which implements something similar to here: https://github.com/facebookresearch/vggsfm/blob/573fdff3f1c730d6be5ffa45c92b1487c4bdb658/vggsfm/models/vggsfm.py#L37.

Alternatively, a class could be defined which only takes in arguments which are serializable, which can inherit from PyTorchModelHubMixin.

Let me know if you need any help

naver-ai / rope-vit

Host models on HF #9

Uploading models