feat(cli): allow users to download models from Kaggle

[KeijiBranshi]

Context

What is the purpose of this PR? Is it to

• [x] add a new feature
• [ ] fix a bug
• [ ] update tests and/or documentation
• [ ] other (please add here)

Please link to any issues this PR addresses.

https://github.com/pytorch/torchtune/issues/1852

Changelog

What are the changes made in this PR?

• 5527fd88a193811fd2c471222be582a0b508e203
  • Added 3 new command line arguments: --source, --kaggle-username, and --kaggle-api-key
  • Splits download logic based on --source value (only valid options are huggingface and kaggle). When --source is not provided, defaults to huggingface
  • Kaggle download logic placed in helper function _download_from_kaggle()
  • The new logic mostly mirrors _download_from_huggingface() with a few notable exceptions:
  • Authentication with kagglehub requires the caller's username and api token as separate arguments to kagglehub.set_kaggle_credentials() (these can also be set as environment variables). Since both are required, appropriate error messages are raised when one is omitted. However, since Kaggle supports anonymous downloads on public non-gated models, the option to omit both is also supported.
  • The exceptions thrown by kagglehub.model_download are different than those raised by huggingface_hub.snapshot_download.
  • kagglehub.model_download doesn't support direct download to a caller-specified directory via --output-dir. This feature is being worked on in kagglehub (see https://github.com/Kaggle/kagglehub/issues/179), but in the meantime a warning was just placed here instead.
  • some extra validation is done on Kaggle model handles (see _validate_kaggle_model_handle) as per the requests made here. This validation is primarily used to raise warnings when downloads may not be guaranteed to work with torchtune.
• a3776d6a7c0d46b1efe5f723e832feb8681a2f57
  • tests were added to validate the CLI behavior when --source kaggle is provided
• b30f741f79c8901ffffd6a02d53e81051a4ae32f
  • updated documentation

Test plan

Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.

• [x] run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
• [x] add unit tests for any new functionality
• [x] update docstrings for any new or updated methods or classes
• [x] run unit tests via pytest tests
• [ ] run recipe tests via pytest tests -m integration_test
  • [x] screenshot as per the instructions in this comment:
• [ ] manually run any new or modified recipes with sufficient proof of correctness
• [ ] include relevant commands and any other artifacts in this summary (pastes of loss curves, eval results, etc.)

UX

If your function changed a public API, please add a dummy example of what the user experience will look like when calling it. Here is a docstring example and a tutorial example

• [ ] I did not change any public API
• [x] I have added an example to docs or docstrings

[pytorch-bot[bot]]

:link: Helpful Links

:test_tube: See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2002

• :page_facing_up: Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

:heavy_exclamation_mark: 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• [DomainsOnly] Jobs fail with GLIBC version not found

:white_check_mark: No Failures

As of commit 00493ef2b493a206b46fa67632962f9e7dce8b99 with merge base d9b6c792f23c7e35dcb4a3a29fee0f30f54f4894 (): :green_heart: Looks good so far! There are no failures yet. :green_heart:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[joecummings]

Awesome stuff!! Will be reviewing this today or tomorrow :)

[KeijiBranshi]

Will wait for CI, but LGTM!

Thanks @KeijiBranshi for your hard work on this and making it easier for torchtune users to download models from the Kaggle Hub!

Thanks so much for your help and review! Really appreciated your guidance on everything.

Looks like some of my test are failing. Could've sworn I ran before pushing, but i guess it slipped through. Resolving those now 🙂

[KeijiBranshi]

@joecummings success!

Thanks again for all your help. Let me know if anything else needs to be done for the release process 🚀

[KeijiBranshi]

@joecummings quick fyi, I just rebased to the most recent main branch commit.

Let us know if there's anything else we should do to get this merged 🙂

[joecummings]

@joecummings quick fyi, I just rebased to the most recent main branch commit.

Let us know if there's anything else we should do to get this merged 🙂

Thanks for the rebase. Will merge this today!! Afterwards, I'll kick off a "nightly" build so users can have immediate access to this feature even before tomorrow's build of torchtune.

[KeijiBranshi]

@joecummings quick fyi, I just rebased to the most recent main branch commit. Let us know if there's anything else we should do to get this merged 🙂

Thanks for the rebase. Will merge this today!! Afterwards, I'll kick off a "nightly" build so users can have immediate access to this feature even before tomorrow's build of torchtune.

Thank you so much!! Looking forward to it.

Please ping me if things break or users see issues 🙂. If I'm unresponsive for any reason, you can also email kaggle-models@google.com.