fireicewolf
commented
5 days ago Dont't delete any json..., that chat_template.json defined the chat format for the llm.
This warning WARNING: LLM user prompt not defined, using default version with wd tags...
Just told you didn't input a user prompt via --llm_user_prompt
It's default value is
Refer to the following words: {wd_tags}. Please describe this image.
If you need to use your own idea in user prompt or system prompt, just write it via --llm_system_prompt and --llm_user_prompt.But keep {wd_tags} in your user prompt , the wd tags will be transfer here after tagging.
specblades
After the captioning with WD, LLM (llama in that case) does not caption the images and the process is complete. That is: the folder contains only captions from WD. (with command
python caption.py D:\DATASETS\equals --model_site modelscope --download_method URL --wd_force_use_cpu --wd_remove_underscore --llm_patch)I fount that in console
2024-10-07 13:35:53,028 - caption.py[line:208] - WARNING: LLM user prompt not defined, using default version with wd tags...Forwarding wd captions to llama system prompt doesnt seem to work.
But after captioning the images with WD, the next attempt will give an error, which disappears if i manually delete chat_template.json in llama\llm folder.
2024-10-07 13:23:25,280 - inference.py[line:230] - INFO: Applying LLM Patch... 2024-10-07 13:23:28,368 - inference.py[line:233] - INFO: LLM Patched. 2024-10-07 13:23:28,383 - inference.py[line:245] - INFO: LLM Loaded in 31.0s. 2024-10-07 13:23:28,383 - inference.py[line:249] - INFO: Loading processor with GPU... Traceback (most recent call last): File "S:\wd-llm-caption-cli\caption.py", line 649, in
my_caption.load_models(get_args)
File "S:\wd-llm-caption-cli\caption.py", line 187, in load_models
self.my_llm.load_model()
File "S:\wd-llm-caption-cli\utils\inference.py", line 250, in load_model
self.llm_processor = AutoProcessor.from_pretrained(self.llm_path)
File "S:\wd-llm-caption-cli\venv\lib\site-packages\transformers\models\auto\processing_auto.py", line 331, in from_pretrained
return PROCESSOR_MAPPING[type(config)].from_pretrained(pretrained_model_name_or_path, kwargs)
File "S:\wd-llm-caption-cli\venv\lib\site-packages\transformers\processing_utils.py", line 916, in from_pretrained
processor_dict, kwargs = cls.get_processor_dict(pretrained_model_name_or_path, kwargs)
File "S:\wd-llm-caption-cli\venv\lib\site-packages\transformers\processing_utils.py", line 660, in get_processor_dict
chat_template = json.loads(text)["chat_template"]
File "C:\Users\capta.pyenv\pyenv-win\versions\3.10.11\lib\json__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\capta.pyenv\pyenv-win\versions\3.10.11\lib\json\decoder.py", line 340, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 4 column 1 (char 5152)