search

approximatelabs / sketch

AI code-writing assistant that understands data content
MIT License
2.24k stars 118 forks source link

LAMBDAPROMPT_BACKEND = StarCoder causes a runtime error #25

Open thinkORo opened 1 year ago

thinkORo commented 1 year ago

When I activate the local execution, I get the following error message: ValueError: The current "device_map" had weights offloaded to the disk. Please provide an "offload_folder" for them. Alternatively, make sure you have "safetensors" installed if the model you are using offers the weights in this format.

To activate the local execution, I did the following:

os.environ['LAMBDAPROMPT_BACKEND'] = 'StarCoder' os.environ['SKETCH_USE_REMOTE_LAMBDAPROMPT'] = 'False' os.environ['HF_ACCESS_TOKEN'] = 'myToken'

Unfortunately, I cannot find a way to set the offload_folder.

How can I do that? Note: "safetensors" is already installed

thinkORo commented 1 year ago

I found a workaround as I didn't find the time to follow the complete execution chain fro Sketch.

I patched the responsive module, modeling_utils.py, and hard-coded the "offload_folder" for the required methods:

  1. _load_state_dict_into_meta_model()
  2. _load_pretrained_model()

But it would be nice not to patch the underlying modules. Therefore, any ideas are welcome.