Add utility function for uploading local folders to huggingface hub

This is useful for uploading any local files to huggingface hub. In particular, I use this for saving dequantized base model weights. Usage is very simple!

from ludwig.utils.hf_utils import upload_folder_to_hfhub
upload_folder_to_hfhub(repo_id="", folder_path="")

This will create a repository on huggingface hub if it doesn't exist, and upload all files in the local folder to huggingface hub at that repo id.

There are a variety of optional parameters that can be specified:

repo_type (str, optional): The type of the repository ('model', 'dataset', or 'space'). Defaults to 'model'.
private (bool, optional): If True, the repository will be private; otherwise, it will be public. Defaults to False.
path_in_repo (str, optional): The relative path within the repository where the folder should be uploaded. Defaults to None, which means the root of the repository.
commit_message (str, optional): A message for the commit associated with the upload.
commit_description (str, optional): A description for the commit associated with the upload.

Here's an example of how I used it:

>>> from ludwig.utils.hf_utils import upload_folder_to_hfhub
>>> upload_folder_to_hfhub(repo_id="arnavgrg/codallama-7b-instruct-nf4-fp16-upscaled", folder_path="/home/ray/codellama-7b-instruct-upscaled")
arnavgrg/codallama-7b-instruct-nf4-fp16-upscaled does not exist. Creating.
Uploading folder /home/ray/codellama-7b-instruct-upscaled to repo arnavgrg/codallama-7b-instruct-nf4-fp16-upscaled.
tokenizer.model: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 500k/500k [00:00<00:00, 1.00MB/s]
model-00003-of-00003.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████| 3.59G/3.59G [01:35<00:00, 37.8MB/s]
model-00001-of-00003.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████| 4.94G/4.94G [02:03<00:00, 40.1MB/s]
model-00002-of-00003.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████| 4.95G/4.95G [02:03<00:00, 40.1MB/s]
Upload 4 LFS files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [02:03<00:00, 30.89s/it]

and the artifacts live here: https://huggingface.co/arnavgrg/codallama-7b-instruct-nf4-fp16-upscaled/tree/main

ludwig-ai / ludwig

Add utility function for uploading local folders to huggingface hub #3805

Unit Test Results