How to inference in the model 'SQLCoder-7B'

[robinji0]

I use the inference code to load ''SQLCoder-7B', it doesn't work. So what't the inference code for 'SQLCoder-7B'? Could you add an example on README?

[rishsriv]

Can you share the code snippet you used to run the model? To loads the 7B param model, you should be changing the code to defog/sqlcoder-7b. Seems to be working for me.

[robinji0]

Can you share the code snippet you used to run the model? To loads the 7B param model, you should be changing the code to defog/sqlcoder-7b. Seems to be working for me.

I use below code to load the model, the path is the local path on my machine. I add the param from_tf=True because the error shows 'If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.'
''' tokenizer = AutoTokenizer.from_pretrained("/workspace/sqlcoder-7b", trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained("/workspace/sqlcoder-7b", trust_remote_code=True, torch_dtype=torch.float16,

load_in_8bit=True,

                                  from_tf=True,
                                  device_map="auto",
                                  use_cache=True)

'''

After I add the param, another error happens: Traceback (most recent call last): model = AutoModelForCausalLM.from_pretrained("/workspace/sqlcoder-7b", File "/home/pai/envs/sqlcoder/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 565, in from_pretrained return model_class.from_pretrained( File "/home/pai/envs/sqlcoder/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3265, in from_pretrained if resolved_archive_file.endswith(".index"): AttributeError: 'list' object has no attribute 'endswith'

[rishsriv]

Ah it seems like you've built huggingface with tensorflow. Mistral-7B (the underlying model for sqlcoder-7b) does not yet work with the tensorflow build.

You'll have to reinstall huggingface with a pytorch build to run it.

[zhouenxian]

I use the inference code to load ''SQLCoder-7B' ,but an error occur. Here are traceback information:

Traceback (most recent call last): File "/datas/work/zex/llm/sqlcoder-7b/spider_sqlcoder_7b.py", line 103, in model = AutoModelForCausalLM.from_pretrained( File "/home/linewell/anaconda3/envs/SQLCoder/lib/python3.9/site-packages/transformers/models/auto/auto_factory.py", line 482, in from_pretrained config, kwargs = AutoConfig.from_pretrained( File "/home/linewell/anaconda3/envs/SQLCoder/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 1022, in from_pretrained config_class = CONFIG_MAPPING[config_dict["model_type"]] File "/home/linewell/anaconda3/envs/SQLCoder/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 723, in getitem raise KeyError(key) KeyError: 'mistral'