bug: how to give trust_remote_code=True option

[jontis]

Describe the bug

I'm trying to start this: openllm start falcon --model-id TheBloke/WizardLM-Uncensored-Falcon-7B-GPTQ --quantize gptq --device 0

It results in: ValueError: Loading this model requires you to execute execute some code in that repo on your local machine. Make sure you have read the code at https://hf.co/... to avoid malicious use, then set the option trust_remote_code=True to remove this error.

openllm, 0.3.6 (compiled: no) Python (CPython) 3.10.12

I'm sure there is an easy way to give this option but I'm not figuring it out. Please advice. Kindly

To reproduce

openllm start falcon --model-id TheBloke/WizardLM-Uncensored-Falcon-7B-GPTQ --quantize gptq --device 0

Logs

..

Environment

$ bentoml env

Environment variable

BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''

System information

bentoml: 1.1.6 python: 3.10.12 platform: Linux-6.4.6-76060406-generic-x86_64-with-glibc2.35 uid_gid: 1000:1000 conda: 22.9.0 in_conda_env: True

conda_packages

### System information (Optional) _No response_

[aarnphm]

You can use TRUST_REMOTE_CODE envvar.

[aarnphm]

falcon should already integrated into transformers, so trust_remote_code might not be needed anymore.

[jontis]

Thank you for your input. I've been trying to set envvar in bash like this: export TRUST_REMOTE_CODE=true

Is that the way to do it?

I've also cloned repo directly from git to make sure I have the latest commits.

It still fails with: OpenLLM$ openllm start falcon --model-id TheBloke/WizardLM-Uncensored-Falcon-7B-GPTQ --quantize gptq --device 0 2023-09-29T11:01:35+0200 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "_service:svc" can be accessed at http://localhost:3000/metrics. 2023-09-29T11:01:35+0200 [INFO] [cli] Starting production HTTP BentoServer from "_service:svc" listening on http://0.0.0.0:3000 (Press CTRL+C to quit) 2023-09-29T11:01:41+0200 [ERROR] [runner:llm-falcon-runner:1] An exception occurred while instantiating runner 'llm-falcon-runner', see details below: 2023-09-29T11:01:41+0200 [ERROR] [runner:llm-falcon-runner:1] Traceback (most recent call last): File "/home/jonathan/mambaforge/lib/python3.10/site-packages/bentoml/_internal/runner/runner.py", line 307, in init_local self._set_handle(LocalRunnerRef) File "/home/jonathan/mambaforge/lib/python3.10/site-packages/bentoml/_internal/runner/runner.py", line 150, in _set_handle runner_handle = handle_class(self, args, kwargs) File "/home/jonathan/mambaforge/lib/python3.10/site-packages/bentoml/_internal/runner/runner_handle/local.py", line 27, in init self._runnable = runner.runnable_class(runner.runnable_init_params) # type: ignore File "/home/jonathan/code/OpenLLM/openllm-python/src/openllm/_llm.py", line 1166, in init if not self.model: raise RuntimeError('Failed to load the model correctly (See traceback above)') File "/home/jonathan/code/OpenLLM/openllm-python/src/openllm/_llm.py", line 748, in model model = self.load_model(self._model_decls, self._model_attrs) File "/home/jonathan/code/OpenLLM/openllm-python/src/openllm/_assign.py", line 71, in inner return fn(self, *decls, *attrs) File "/home/jonathan/code/OpenLLM/openllm-python/src/openllm/serialisation/init.py", line 75, in caller return getattr(importlib.import_module(f'.{serde}', name), fn)(llm, args, kwargs) File "/home/jonathan/code/OpenLLM/openllm-python/src/openllm/serialisation/transformers/init.py", line 158, in load_model config, hub_attrs, attrs = process_config(llm.model_id, llm.trust_remote_code, attrs) File "/home/jonathan/code/OpenLLM/openllm-python/src/openllm/serialisation/transformers/_helpers.py", line 42, in process_config config, attrs = transformers.AutoConfig.from_pretrained(model_id, return_unused_kwargs=True, trust_remote_code=trust_remote_code, hub_attrs, **copied_attrs) File "/home/jonathan/mambaforge/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1026, in from_pretrained trust_remote_code = resolve_trust_remote_code( File "/home/jonathan/mambaforge/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 618, in resolve_trust_remote_code raise ValueError( ValueError: Loading TheBloke/WizardLM-Uncensored-Falcon-7B-GPTQ requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option trust_remote_code=True to remove this error.

[aarnphm]

You can just try TRUST_REMOTE_CODE=true openllm start .... Usually you shouldn't export the envvar.

I will take a look at this.

[aarnphm]

Hi there can u try with the new version of openllm?

[aarnphm]

In think this has been addressed. I have tested this with Phi models