RFC Creating a coherent user and developer experience regarding integrations

adrinjalali commented 2 years ago

Status-quo

At the moment, there are different ways to interact with the hub. This RFC aims to look at a summary of them, and try to find ways to enhance end users' and developers' experience.

Other than the website and git, right now we provide a set of tools available in huggingface_hub which users and third party developers can use to interact with the hub.

Some integration related interfaces that exist as of writing this RFC are listed bellow. This is not necessarily an exhaustive list, it is only for us to see what the ecosystem at the moment looks like.

huggingface_hub:

# huggingface_hub.hf_api
HfApi:
    whoami(...)
    {set, unset}_access_token(...)
    get_{model, dataset}_tags(...)
    list_{models, datasets, metics, repo_files}(...)
    {model, dataset}_info(...)
    {create, delete, move}_repo(...)
    update_repo_visibility(...)
    upload_files(...)
    delete_file(...)
    get_full_repo_name(...)

# huggingface_hub.file_download
cached_download(...)
hf_hub_download(...)
hf_hub_url(...)

# huggingface_hub.snapshot_download
snapshot_download

# huggingfaace_hub.hub_mixin:

ModelHubMixin:
    save_pretrained(..)
    from_pretrained(...)

PyTorchModelHubMixin(ModelHubMixin):
    ...

# huggingface_hub.keras_mixin:

KerasModelHubMixin(ModelHubMixin):
    ...

save_pretrained_keras(...)
from_pretrained_keras(...)
    return KerasModelHubMixin...
push_to_hub_keras(...)

# huggingface_hub.fastai_utils
# proposed in https://github.com/huggingface/huggingface_hub/pull/678

from_pretrained_fastai(...)
push_to_hub_fastai(...)
save_fastai_learner(...)

And the CLI pattern:

$ huggingface-cli --help
usage: huggingface-cli <command> [<args>]

positional arguments:
  {login,whoami,logout,repo,lfs-enable-largefiles,lfs-multipart-upload}
                        huggingface-cli command helpers
    login               Log in using the same credentials as on huggingface.co
    whoami              Find out which huggingface.co account you are logged in as.
    logout              Log out
    repo                {create, ls-files} Commands to interact with your huggingface.co repos.
    lfs-enable-largefiles
                        Configure your repository to enable upload of files > 5GB.
    lfs-multipart-upload
                        Command will get called by git-lfs, do not call it directly.

transformers

The following pattern is available for all transformers models:

AutoModel:
    load_repo_checkpoint(...)
    save_pretrained(..., push_to_hub: bool)
    from_pretrained(...)
    push_to_hub(...)

Some of the above methods may not be available for all models.

We have a documentation page here with more ways to work with the hub, such as:

For transformers, TrainingArguments accepts push_to_hub=True, which means the model will be uploaded to the hub as it's being trained.
TensorFlow users can use PushToHubCallback to achieve the same thing.

Note that the docs are not clear about when the upload happens and how often.

spacy

spacy developers provide a spacy-huggingface-hub library, which provides:

# spacy_huggingface_hub:
push(...)

And the following CLI pattern:

huggingface-cli login
python -m spacy package ./en_ner_fashion ./output --build wheel
cd ./output/en_ner_fashion-0.0.0/dist
python -m spacy huggingface-hub push en_ner_fashion-0.0.0-py3-none-any.whl

allennlp

Natively available in the library, there is:

# allennlp.commands.push_to_hf
push(...)
# loading
Predictor:
    from_path(...)

They also natively provide a CLI sub-command as:

allennlp push-to-hf ...

adapterhub

AutoAdapterModel class from transformers library has methods to push and load Adapter models from Hugging Face Hub:

AutoAdapterModel:
    from_pretrained(...)
    load_adapter(...)
    push_adapter_to_hub(...)

sentence-transformers

sentence-transformers library has model pushing and loading functionalities inside model class itself.

SentenceTransformer:
    save_to_hub(...)
    __init__(...) #uses snapshot download when initializing model

stablebaselines3

For stablebaselines3, thanks to @simoninithomas, we have developed huggingface-sb3 which provides load and push:

# huggingface_sb3
load_from_hub(...)
push_to_hub(...)

asteroid

BaseModel class of asteroid library has from_pretrained method for loading from Hugging Face Hub.

BaseModel:
    from_pretrained(...)

They also mention that users can directly use git as:

git clone model_repo_url
git add . && git commit -m "message"
git push

espnet

espnet allows models to be uploaded while/after being trained through CLI:

./run.sh --stage 8 --skip-upload-hf false --hf-repo <my_repo>

Usage, different perspective

From the users' perspective, there are a few different ways to interact with the hub, independent of what library they use.

The website (hf.co)

The website enables users to browse models, datasets, and spaces. Users can also test the output of many models directly from the website if the tasks of the model are supported or if there is a Space which by design is interactive.

Visiting a model on the website, points users to these ways to work locally with the model:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-zh-en")

model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-zh-en")

Or using git:

git lfs install
git clone https://huggingface.co/Helsinki-NLP/opus-mt-zh-en

# if you want to clone without large files – just their pointers
# prepend your git clone with the following env var:
GIT_LFS_SKIP_SMUDGE=1

Note that we do not tell users what to do once they've cloned the repo. Running the python code after cloning will still download the model into cache before loading; it does not use the downloaded model.

REST API

Users can directly work with the hub using the REST endpoints documented here. Users can also subscribe to certain webhooks if they wish to get notified about certain events on the hub.

Git

A user could potentially work with the hub w/o any installed hub related dependency other than git and git-lfs since one can treat hub repos as a git repo. If one knows which files are to be in a repo and how to write a model card as a README, all one needs is git to interact with the hub to create and update repos.

CLI

One way to work with the hub is using the command line interface. huggingface_hub provides one as huggingface-cli explained above, and other libraries such as spacy also expose their own CLI which users can use to interact with the hub.

huggignface-cli does not provide any functionality specific to any integration at the moment.

If the third party library decides to provide CLI tools for interacting with the hub, at the moment, it doesn't add to huggingface-cli and instead introduces its own CLI. For instance, installing spacy-huggingface-hub adds huggingface-hub to spacy as : $ spacy huggingface-hub ...

huggingface_hub python library

The core of huggingface_hub python library is a combination of being a python wrapper around these API endpoints and using git in the background for certain tasks.

Users can choose to use huggingface_hub python library to interact with the hub. The core of the library provides tools to interact with the hub and is the go-to place for users if they wish to use python to list, create, move, delete, or update repositories in the hub.

The library also provides integrations for a few third party frameworks, namely Keras and Pytorch, and fastai in the making. Those integrations provide a few functions to push to and pull from the hub.

Other libraries, using python

Some other frameworks provide certain functionalities to work with the hub using python, and is either implemented inside their main library or as a third library.

Directly using the ML library

Certain libraries, such as transformers, natively support certain integrations with the hub. For instance, all transformers models support pushing to the hub using save_pretrained(..., push_to_hub=True). The library can also choose to support {down}loading from the hub using code such as:

    AutoTokenizer.from_pretrained(...)

Using a third library specific to an ML library

For certain libraries, such as stablebaselines3, there is a third library, huggingface-sb3 in this example, which provides certain functionality for users to work with the hub. E.g:

# huggingface_sb3
load_from_hub(...)
push_to_hub(...)

Note that there are no standard names or ways to implement those integrations. Some libraries choose to have them as class or instance methods, and some are provided as a function under a certain namespace.

Proposal - Users' Perspective

To make users' experience consistent across different frameworks, we can look at each way users interact with the hub and see what we can do to improve their experience. We should also decide which are the ways we'd like to recommend to people and nudge them in that direction. The action items concluded from this process can be implemented by developers maintaining that area.

Git

We want to keep encouraging people to use pure git to interact with the hub. However, there are things we can do to make their lives easier:

The interface at hf.co can be more explicit about how users can load and use the model once they've cloned it using git. At the moment the website tells users how to clone a repo, but doesn't tell them how to use those downloaded files.
To create a repository, users can simply create a repo and upload whatever they want to it. However, if they want the widgets and inference to work correctly, they need to satisfy certain requirements, such as having a pyproject.toml with mentioned dependencies, etc. Since people who are working on widgets and the inference backend are always involved in accepting and setting those specs, it would make sense for it to be a place on our docs explaining requirements for each library (somewhere under hf.co).

CLI

There are two ways we can see users interacting with the hub through CLI. One is by using huggingface-cli, and the other is by using whatever tool the third party developers provide.

Ideally, we would like to recommend users to use huggingface-cli for all interactions without having the implementation details for each library in the scope of huggingface_hub. For this, we can use dispatching mechanisms the same way as it's done in spacy and spacy-huggingface-hub. We would define a standard interface to be implemented for defining different commands, such as push, pull, etc., and delegate that to the corresponding library. From users' perspective, it would look like:

huggingface-cli spacy push ...
huggingface-cli transformer pull ...
huggingface-cli fastai create ...

Under the hood we use the corresponding library, and if not installed, we ask them to install that dependency.

The desired and expected flow and supported commands and their specs shall be discussed in a separate issue.

Python

If users are interacting with the hub from within python, we can give them a coherent interface without including the implementation inside the library.

from huggingface_hub.spacy import from_pretrained
from huggingface_hub.transformers import push_to_hub

The above code would load the appropriate functions from third party libraries. What we need to do for that to happen is to define the dispatch interfaces and the API.

Since each function will be implemented in and loaded from different places, we don't have to care too much about their signature. What we need to do is to create a flow with required functions and their specs so that users get a coherent experience across the board.

Libraries can still choose to implement integrations as class/instance methods of their corresponding classes, but we don't necessarily encourage that.

There shall be another issue to discuss the details of this section.

Proposal - Developer's Perspective

We shall develop the specs discussed above and include the base implementations inside huggingface_hub. Third party developers can choose to implement them inside their library, or to have a separate library dedicate to hug's integrations.

For the latter case, we also create a template repo where we include all which is needed to create a package for this purpose, including all the setup and CI related files. Developers would then need to instantiate from that template repo, and fill in the blanks. They are of course free to implement more than the minimum requirement we define in the specifications.

Acknowledgements: This RFC has been developed with a ton of help from @merveenoyan , @osanseviero , and @LysandreJik

LysandreJik commented 2 years ago

Thanks for the write-up! I agree that:

Giving the libraries owners the tools to integrate upload/download support in their library so that it is coherent with their own API
Offering a very simple integration so that all libraries can then easily share a similar CLI API on top of their own

Are great ways to move forward and to ensure simplicity of usage for users. Excited to see this taking shape.

osanseviero commented 2 years ago

Very nice write-up! Thank you! 🚀 It's also nice to see all this info on libraries in a single place.

The interface at hf.co can be more explicit about how users can load and use the model once they've cloned it using git. At the moment the website tells users how to clone a repo, but doesn't tell them how to use those downloaded files.

This is a good point. I think most users don't even care at all about cloning the repo, since most times they just want to use the library to load the repo. This is useful feedback and maybe we should open an issue in moon landing to further discuss this . cc @gary149 @julien-c

Since people who are working on widgets and the inference backend are always involved in accepting and setting those specs, it would make sense for it to be a place on our docs explaining requirements for each library (somewhere under hf.co).

This is happening with https://github.com/huggingface/hub-docs/issues/62 🚀 🔥 we aim to launch this in 2-3 weeks.

About CLI

This seems as an interesting approach! IMO though most of the time what people do in a library is use some save_model function to export the pickle/whatever format file. So a user would then need to dohuggingface-cli transformers push ...whl repo_name or something like that, but as a user, I would likely want to avoid switching from code to CLI, so doing trainer.push_to_hub feels much more natural to me. I can see how this can be useful for certain use cases such as spacy in which the users use the CLI to package their models, but in terms of developer usage I don't think this will be a widely adopted approach.

(I'm also curious what create or pull would do in this context, as pull would be the same across all libraries I think)

Giving the libraries owners the tools to integrate upload/download support in their library so that it is coherent with their own API

💯

osanseviero commented 2 years ago

cc @patrickvonplaten who has done lots of integrations within audio libraries and whose input would be valuable here too 😄

patrickvonplaten commented 2 years ago

We want to keep encouraging people to use pure git to interact with the hub. However, there are things we can do to make their lives easier:

The interface at hf.co can be more explicit about how users can load and use the model once they've cloned it using git. At the moment the website tells users how to clone a repo, but doesn't tell them how to use those downloaded files.

+1 Agree

There are two ways we can see users interacting with the hub through CLI. One is by using huggingface-cli, and the other is by using whatever tool the third party developers provide.

Ideally, we would like to recommend users to use huggingface-cli for all interactions without having the implementation details for each library in the scope of huggingface_hub. For this, we can use dispatching mechanisms the same way as it's done in spacy and spacy-huggingface-hub. We would define a standard interface to be implemented for defining different commands, such as push, pull, etc., and delegate that to the corresponding library. From users' perspective, it would look like:

huggingface-cli spacy push ... huggingface-cli transformer pull ... huggingface-cli fastai create ... Under the hood we use the corresponding library, and if not installed, we ask them to install that dependency.

The desired and expected flow and supported commands and their specs shall be discussed in a separate issue.

Hmm does this fit though with the general design of huggingface-cli being able to create a model repo:

huggingface-cli repo create

and login, etc...

Personally, I think it'd be more important to first add functionality allowing users to not just create model repos but also dataset repos and space repos

from huggingface_hub.spacy import from_pretrained from huggingface_hub.transformers import push_to_hub

From a first intuitiion this looks a bit weird to me because for me libraries like transformers wrap huggingface_hub and not the other way around. Also push_to_hub is usually differentiated by different classes in transformers like PreTrainedModel, PretrainedTokenizer -> how would this fit in the API here?

adrinjalali commented 2 years ago

Hmm does this fit though with the general design of huggingface-cli being able to create a model repo:
huggingface-cli repo create
and login, etc...

I don't think we want to remove the top level commands, but certain integrations would need specialized commands. For instance, if my library needs to include certain metadata about the environment under which the model can run, the create command can take care of that. (not attached to any of these names, we could call it something else).

Personally, I think it'd be more important to first add functionality allowing users to not just create model repos but also dataset repos and space repos

Some of that already exists, but we need to better document them.

From a first intuitiion this looks a bit weird to me because for me libraries like transformers wrap huggingface_hub and not the other way around. Also push_to_hub is usually differentiated by different classes in transformers like PreTrainedModel, PretrainedTokenizer -> how would this fit in the API here?

Those implementations are inside those libraries, not inside huggingface_hub. In this example, the implementation in transformers would know which method to call to save the model. The from huggingface_hub.transformers import push_to_hub would only be a shorthand import for importing what's needed to be used from transformers itself.

huggingface / huggingface_hub