Looking into the huggingface model updates, it looks like any substantial model update is created as a new model (different id) on huggingface. Any significant model revisions would require us changing the model id and would probably need to be a manual process. Based on that, instead of checking for updated models, I wrote this code that checks if the models we need are saved in the right directory and downloads them if not. All models are imported from the same location. The models are put in the gitignore but they will install when the pipeline is run on the server the first time.
Looking into the huggingface model updates, it looks like any substantial model update is created as a new model (different id) on huggingface. Any significant model revisions would require us changing the model id and would probably need to be a manual process. Based on that, instead of checking for updated models, I wrote this code that checks if the models we need are saved in the right directory and downloads them if not. All models are imported from the same location. The models are put in the gitignore but they will install when the pipeline is run on the server the first time.