Closed MarleneKress79789 closed 5 months ago
Please add some explanation for the Model Uploader Script. In particular we need to mention that the --local-model-path
needs to point to a cache directory. For example, one may download a model using this code:
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained(<MODEL-NAME>, cache_dir="whatever")
model = AutoModelForSequenceClassification.from_pretrained(<MODEL-NAME>,, cache_dir="whatever")
Then it can be uploaded to the BucketFS with this command:
python -m exasol_transformers_extension.upload_model \
...
--local-model-path whatever
Maybe it's worth mentioning that to use TE_MODEL_DOWNLOADER_UDF the DB must have access to the internet. Currently this is not the case with the Docker DB by default. One needs to specify a name server. For example --nameserver 8.8.8.8
will set it to use Google DNS.