Open SouLeo opened 4 months ago
Hey sorry https://github.com/Muennighoff/mteb.git@offlineaccess
is very outdated at this point. I recommend you to just use the regular MTEB and make the necessary changes to make it work offline (which would be similar to the changes made in https://github.com/Muennighoff/mteb.git@offlineaccess). I think it should be very straightforward. It would be amazing if you could open a PR to allow offline evaluation, too, so we have it in the main mteb. 🙌
Hello, I'm trying to run MTEB on a cluster without internet access, but I am struggling. Here are the following instructions I've followed:
$ pip install mteb
$ !pip install --upgrade git+https://github.com/Muennighoff/mteb.git@offlineaccess
$ git clone git@hf.co:datasets/mteb/banking77
I then run:
with the following environment variables set in my .bashrc:
But I get this error:
Error while evaluating Banking77Classification: Couldn't find a dataset script at <path to data>/mteb/banking77/banking77.py or any data file in the same directory.
I'm not sure where I obtain the script:
banking77.py
or if I am properly following the offline evaluation instructions. Any help would be immensely appreciated.