Add support for validating whether an ONNX model exists in a repo's subdirectory

gregfurman commented 3 months ago

The huggingface downloader does not allow for specifying a subdirectory when validating a model and tokenizer.

So in the example of distilbert/distilbert-base-uncased-finetuned-sst-2-english there is an onnx subdirectory that contains all the files of interest but the validation fails since this is not in the root.

Not sure how best to do this but you could look at:

Extending validateDownloadHfModel to allow for including subdirectories -- currently you can inject this into the branch value but there should probably be an optional variable to include this https://github.com/knights-analytics/hugot/blob/fd59728f69cd76fb3e011a89929de441edd7bc52/downloader.go#L84-L87
Extending checkURL to be able to validate within a subdirectory (see API output) https://github.com/knights-analytics/hugot/blob/fd59728f69cd76fb3e011a89929de441edd7bc52/downloader.go#L124-L127

riccardopinosio commented 3 months ago

I think this might be a bug as the validator should already accept subfolders, will investigate

riccardopinosio commented 3 months ago

Indeed was a bug, thanks for catching it! I added a fix and a test for it, will be in the next release.

knights-analytics / hugot