Open linuxmagic-mp opened 1 year ago
linuxmagic-mp
commented
1 year ago Okay, the problem here appears to lie in the way AutoModelForCausalLM is called.. When using a ctransformers method, we should be calling it with a model_file argument. See https://github.com/marella/ctransformers/issues/60. Otherwise, it can return any .bin from the path, including training_args.bin, which of course will fail ggml checks.
linuxmagic-mp
commented
1 year ago And to continue, I have the actual filename in the reference file
cat ~/lollms_data/models/llama_cpp_official/falcon-40b-ggml.reference
~/models/WizardLM-Uncensored-Falcon-40b/ggml-model-falcon-40b-wizardlm-qt_k5.binHoweer, it doesn't see that it properly sees that as a file. Instead, earlier that was parsed down to model_path, and model_name was not preserved.. So when we run.. _find_model_path_from_dir, the optional filename is not passed in. AutoModelForCausalLM.from_pretrained(), which was called after get_model_path
if self.config.model_name.endswith(".reference"):
with open(str(self.lollms_paths.personal_models_path / f"{self.binding_folder_name}/{self.config.model_name}"), 'r') as f:
model_path = Path(f.read()That means if you have the file name in a .reference it will not work. ?
linuxmagic-mp
commented
1 year ago Well, guess I am still talking to myself.. ;) Went up and down this code, and there doesn't seem to be a 'recommended' way to handle setting the actual filename from the config, for the ctransformers binding (or any binding). I think the work is all about pulling down the needed models, but when you have a directory of different quantized models you are kind of hooped.
The transformers binding 'should' be doing more that the simple get_model_path(self), which in itself could use a little validation.It should only read the .reference file once if it exists, and set both path and filename if the .reference file contains a fullpath to a model.
Not really sure why the transformer bindings were decided that they should NOT be part of the library itself, and placed in the users lollms personal settings.. Seems like a strange design choice.
/c_transformers/init.py needs to be updated. .
To get on with things, since noone is commenting, no use making a pull request.. I just hacked get_model_path() to set self.model_file, and the zoo.. init.py, to pass that in when it calls AutoModelForCasual. NOTE, this would ALSO work if get_model_path could return whatever is in the .reference file, but that goes against my grain, as the function could be used elsewhere, where it really does need the 'path'
ParisNeo
commented
1 year ago Hi there, I am sorry. I has loads of issues opened and yours was down and I was both in vacation, then was coding so i didn't see it. I have dixed your problem. It was just about the code that determines the type of the file out of its name. Now Wizard stuff is considered llama so all of his code is now accepted as llama.
On Ubuntu Focal..
I have both the original, as well as the converted, and quantized versions in that directory.
Easy fix? Suggest that when the 'path' is chosen, if it doesn't see a GGML file, that maybe it can prompt for the actual name of the GGML file of choice.
It isn't clear from the errors, but it is actually bubbling up from ctransformers, and have filed ticket there as well.
Issues with ctransformers_llm_create and Falcon-40b model #60
Only posting here to see if perhaps it is not called with correct parameters, but it 'appears' this isn't a problem with that.
Ticketing here for others trying to use ctransformers with Falcon for now.