exasol / transformers-extension

An Exasol extension for using state-of-the-art pretrained machine learning models via the Hugging Face Transformers API.

MIT License

2 stars 2 forks source link

Support Transformers >= 4.22 #42

Closed umitbuyuksahin closed 1 year ago

umitbuyuksahin commented 1 year ago

Background

The directory structure where pre-trained models are downloaded has been changed with transformers 4.22
While the models files directly are downloaded into the cache directory in older versions, with version 4.22 transformers uses complex directory structure.
We need to upload this directory to bucket as we downloaded.
In order to complete this issue, the ticket https://github.com/exasol/bucketfs-python/issues/1 must be done.

Acceptance Criteria

switched to upload_directory method in bucketfs-utils
update toml file with `transformers="^4.22"'
- pay attention that newer versions install nvidia modules which increases SLC size ~2GB

redcatbear commented 1 year ago

Needs another review from Marlene.