Hugging Face Hub integration

[qubvel]

Hi @siyuanliii,

Thanks for this great work! I am excited for the code release!

I am reaching you about the integration with the 🤗 hub from day 0, so you can track the download number, have a nice model card and anyone can automatically load the model using from_pretrained (and push it using push_to_hub, similar to models in the Transformers library). It leverages the PyTorchModelHubMixin class which allows to inherit these methods.

Usage can be as follows:

from masa.models import MasaModel

model = MasaModel.from_pretrained("<your-username-on-hf-hub>/<model-name>")

This means people don't need to manually download a checkpoint first in their local environment, it just loads it automatically from the hub. The safetensors format is used to ensure safe serialization of the weights rather than pickle.

Integration usually is really simple, you just need to subclass your PyTorch model class from nn.Module together with PyTorchModelHubMixin class. For example:

import torch
import torch.nn as nn
+ from huggingface_hub import PyTorchModelHubMixin

class MasaModel(
        nn.Module,
+        PyTorchModelHubMixin,
+        library_name="masa",
+        repo_url="https://github.com/siyuanliii/masa",
        # ^ Optional metadata to generate model card
    ):
    def __init__(self, ...):
        ...

Then, you will be able to push the pretrained model to the hub to make it discoverable:

from masa.models import MasaModel

model = MasaModel()
model.save_pretrained("<your-username-on-hf-hub>/<model-name>", push_to_hub=True)

Would you consider this integration? Please, let me know if you have any questions or concerns!

Thank you, Pavel Iakubovskii ML Engineer @ HF 🤗

[siyuanliii]

Hi Pavel, thanks for reaching out and being interested in our work. Yes, I'd love to integrate the model weights into the hugging face hub!

[qubvel]

Awesome, let me know if you have any questions! Just ping me in this issue 🤗

[ZhenyuSun-Walker]

Hi qubvel, would you mind telling me that in which file should I write the code that introduce the model from hug to run the whole project?

[qubvel]

Hi @SorryGeats, sorry didn't get the question. Do you want integration with HF hub for easy checkpoint downloads?

[qubvel]

The project looks interesting and would be really nice to have it natively integrated with HF transformers, just in case you are interested in such an integration just let me know! By integration I mean adding this model in transformers package. Looking forward to your reply, thanks! @siyuanliii

[siyuanliii]

@qubvel Thank you very much! I'd like such an integration. I see that HF transformers already have many strong detectors integrated to provide detections. Then, we can safely remove the mmdetection dependencies. MASA in its core is an appearance model. For example, the independent MASA-R50 is just a standard ResNet-50. Providing bounding boxes from any detectors, we can associate them using MASA's features. I'd say integrated MASA-R50 is a good starting point. MASA unified version, such as the MASA-GroundingDINO, reuses GroundingDINO's backbone features, which needs additional modification to HF transformers' existing GroundingDINO code.

[qubvel]

@siyuanliii Object detection models in transformers return their hidden states that can be used as features to apply masa_adapter, of course, it has to be checked if it matches exactly with those produced in your implementation. So, I'm curious did you explore the possibility of reusing transformers grounding dino or there is something specific that has to be modified

[siyuanliii]

ok, that would make the integration work easier! No, I haven't tried any detectors in transformers. MASA needs the backbone features in different stages. If transformers can return that, then I don't see we need any modification..

[rolson24]

@siyuanliii, @qubvel Would love to help out with this!