search

facebookresearch / jepa

PyTorch code and models for V-JEPA self-supervised learning from video.
Other
2.68k stars 254 forks source link

Add Hugging Face discoverability #42

Closed NielsRogge closed 7 months ago

NielsRogge commented 8 months ago

Hi @MidoAssran and team, thanks for this great model!

Currently models aren't easily discoverable, there are no JEPA models on the 🤗 hub, so this PR proposes to leverage the PyTorchModelHubMixin which is a minimal class to add from_pretrained and push_to_hub capabilities to any custom nn.Module.

All you need to do is inherit from this class, and then you can do:

from src.models.vision_transformer import VisionTransformer

model = VisionTransformer.from_pretrained("nielsr/vit-large-patch16-v-jepa")

This also comes with automated download metrics, meaning you will be able to see how many times people use each of the various JEPA models.

The model is here for now: https://huggingface.co/nielsr/vit-large-patch16-v-jepa, but we could push and move all checkpoints to the Meta organization on the 🤗 hub.

Here's a notebook to showcase it end-to-end.

Kind regards,

Niels ML @ HF

NielsRogge commented 7 months ago

Hi @MidoAssran would there be any interest in this?

We'd love to see checkpoints shared on HF 🤗