search

UKPLab / sentence-transformers

State-of-the-Art Text Embeddings
https://www.sbert.net
Apache License 2.0
15.34k stars 2.48k forks source link

Issue loading remote source code offline? #3043

Open brendanartley opened 1 week ago

brendanartley commented 1 week ago

Hi there,

I am trying to load the nomic-ai/nomic-embed-text-v1 onto a machine with internet disabled. It seems a similar issue was mentioned here.

I am able to save/load variants such as all-mpnet-base-v2 and all-MiniLM-L6-v2 just fine, but I believe the remote source code is causing an issue.

First, I save the model on a machine with internet enabled.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("nomic-ai/nomic-embed-text-v1", trust_remote_code=True).cuda()
model.save("./nomic-embed-text-v1.pt")

Next, I attempt to load the model on a machine with internet disabled.

embedding_model = SentenceTransformer(
    "nomic-ai/nomic-embed-text-v1", 
    trust_remote_code=True,
    local_files_only=True,
).cuda()
embedding_model.load("./nomic-embed-text-v1.pt")

This results in the error below.

Could not locate the configuration_hf_nomic_bert.py inside nomic-ai/nomic-bert-2048.

Is it possible to load models w/ remote source code given these constraints?

tomaarsen commented 2 days ago

Hello!

It is possible, but you have to make some modifications. Because the remote model itself requires modeling files that are in another remote location, we can't use local_files_only=True to only use cached files, as although we'd use the local files from the model itself, it would still require downloading the modeling files from another remote location.

Here are the steps to get it working locally without internet:

  1. Save the model locally:

    from sentence_transformers import SentenceTransformer

model = SentenceTransformer("nomic-ai/nomic-embed-text-v1", trust_remote_code=True)
model.save_pretrained("nomic-embed-text-v1-local")
  2. Observe the config.json of the downloaded model and search for auto_map. This will be a mapping of autoclass names to either {repository}--{file}.{class} or {file}.{class}. Either way, we want to download all files mentioned in this mapping. Place them in the local model repository.
  3. If the config.json auto_map has the format of {repository}--{file}.{class}, update it to {file}.{class} instead. We have the files in the same directory as the config now, so this should work.
  4. Load the model "like normal", but with local_files_only=True: 
    embedding_model = SentenceTransformer(
   "nomic-embed-text-v1-local", 
   trust_remote_code=True,
   local_files_only=True,
)

    P.s. I'm not sure if trust_remote_code=True is still necessary - it might not be, feel free to experiment.