Where to get the *.bin models from?

[theelefant]

I've checked the models provided by TheBloke, and they are all in the safetensor format. It appears that I cannot use safetensors with your nodes. Any help?

I tried ggml-vic13b-q5_1.bin gpt-x-alpaca-13b-native-4bit-128g-cuda.pt koala-7B.ggmlv3.q4_0.bin stable-vicuna-13B-GPTQ-4bit.compat.no-act-order.safetensors vicuna-13b-4bit-128g.safetensors

Only the *.bin models are shown in the GPT Loader node. But they return an error like this:

gguf_init_from_file: invalid magic number 67676a74
error loading model: llama_model_loader: failed to load model from D:\stable-diffusion-webui_AUTOMATIC1111\stable-diffusion-webui-master\models\GPTcheckpoints\koala-7B.ggmlv3.q4_0.bin

llama_load_model_from_file: failed to load model
!!! Exception during processing !!!
Traceback (most recent call last):
  File "D:\ComfyUI\ComfyUI\execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "D:\ComfyUI\ComfyUI\execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "D:\ComfyUI\ComfyUI\execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "D:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-N-Nodes\py\gptcpp_node.py", line 50, in load_gpt_checkpoint
    llm = Llama(model_path=ckpt_path,n_gpu_layers=gpu_layers,verbose=False,n_threads=n_threads, n_ctx=4000, )
  File "D:\ComfyUI\ComfyUI\comfyUI_env\lib\site-packages\llama_cpp\llama.py", line 323, in __init__
    assert self.model is not None
AssertionError

[Nuked88]

Uhm you should find a lot of ggml model in the thebloke repository, I'm using models like TheBloke/Nous-Hermes-Llama-2-7B-GGML or other GGML model based, if the error is "invalid magic number" the technology behind the model you are using is probably not supported by llama-cpp-python (maybe you need to update it relaunching install_dependency.bat) or it's corrupted.Let me know if you solved the issue!

[theelefant]

Thank you for getting back to me. I've made some adjustments to try and get your nodes functioning properly, but I have encountered an new error and I am currently unable to progress further:

_got prompt Model Path D:\stable-diffusion-webui_AUTOMATIC1111\stable-diffusion-webui-master\models\GPTcheckpoints\nous-hermes-llama-2-7b.ggmlv3.q5_1.bin !!! Exception during processing !!! Traceback (most recent call last): File "D:\ComfyUI\ComfyUI\execution.py", line 151, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "D:\ComfyUI\ComfyUI\execution.py", line 81, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "D:\ComfyUI\ComfyUI\execution.py", line 74, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "D:\ComfyUI\ComfyUI\custom_nodes\ComfyUI-N-Nodes\py\gptcpp_node.py", line 50, in load_gpt_checkpoint llm = Llama(model_path=ckpt_path,n_gpu_layers=gpu_layers,verbose=False,n_threads=n_threads, n_ctx=4000, ) File "D:\ComfyUI\ComfyUI\comfyUI_env\lib\site-packages\llamacpp\llama.py", line 323, in init assert self.model is not None AssertionError

If I'm interpreting this error correctly, it seems that LLAMA cannot locate the specified model, which is quite strange because the path appears to be correct. I'm able to select the model through your GPT Loader Node. I've even attempted using all known iterations of "/", "\", and "//" in the path, but none of these approaches seem to work. So, it looks like I've hit a bit of a wall at the moment.

[Nuked88]

Hi, i did some digging and i found this: https://github.com/abetlen/llama-cpp-python/issues/643 Apparently in the last update llama.cpp dropped the support for all the ggmlv3 models in favor of the new gguf format... If you want to use the ggmlv3 models you need to go back to v0.1.79

Do a git pull in the ComfyUI-N-Nodes folder for update this repo at the latest version and launch install_dependency_ggml_models.bat and see if it works, or just download a GGUF model

[boricuapab]

The .safetensor LLMs wont work with llama cpp python as those are gptq versions.

If you have llama cpp standalone installed though you could convert the ggml v3 files you have into gguf if they haven't been done so yet by The Bloke.

[theelefant]

Gentlemen, thank you very much! Converting to gguf did the trick.

[pottingerb]

Where are the gguf files?

[theelefant]

Where are the gguf files?

You can find some here: https://huggingface.co/models?search=gguf

Alternatively, if you already have models in the GGMLv3 format and have llama.cpp installed on your system, you can convert them to GGUF using the script "convert-llama-ggmlv3-to-gguf.py" from llama.cpp.