Closed hamishcunningham closed 1 year ago
In wisng everything in chatbot/
is no longer required and should be removed (just did) - it hasn't been required or updated in a long time so there's no telling where this is coming from there. The issue you referenced seems to (from a quick read) be related to Tensorflow.
If you want to use LLM support you can copy settings.py to custom_settings.py and set support_chatbot: True. You can change the chatbot model path and basename to a local path or anything from HuggingFace that's GPTQ LLaMa. As you can tell we have a LOT of documentation updates to do :).
Generally I'm not surprised by this - protobuf and grpcio have long been the source of my biggest dependency problems with python in this space but let us know if you continue to run into this anywhere.
Tnx @kristiankielhofner !
I tried this in custom_settings.py
# Path to chatbot model - download from HuggingFace at runtime by default (gets cached)
# chatbot_model_path: str = 'TheBloke/vicuna-13b-v1.3-GPTQ'
chatbot_model_path: str = 'models/vicuna'
# Chatbot model basename
# chatbot_model_basename: str = 'vicuna-13b-v1.3-GPTQ-4bit-128g.no-act.order'
chatbot_model_basename: str = 'gptq_model-4bit-128g'
and got these warnings:
willow-inference-server-wis-1 | [2023-07-04 07:41:00 +0000] [90] [INFO] CHATBOT: Using model models/vicuna and CUDA, attempting load (this takes a while)...
willow-inference-server-wis-1 | The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.
willow-inference-server-wis-1 | The safetensors archive passed at models/vicuna/gptq_model-4bit-128g.safetensors does not contain metadata. Make sure to save your model with the `save_pretrained` method. Defaulting to 'pt' metadata.
willow-inference-server-wis-1 | [2023-07-04 07:41:29 +0000] [90] [INFO] Warming models...
willow-inference-server-wis-1 | [2023-07-04 07:41:34 +0000] [90] [INFO] Warming chatbot... This takes a while on first run.
willow-inference-server-wis-1 | Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers
willow-inference-server-wis-1 | pip install xformers.
willow-inference-server-wis-1 | The model 'LlamaGPTQForCausalLM' is not supported for text-generation. Supported models are ['BartForCausalLM', 'BertLMHeadModel', 'BertGenerationDecoder', 'BigBirdForCausalLM', 'BigBirdPegasusForCausalLM', 'BioGptForCausalLM', 'BlenderbotForCausalLM', 'BlenderbotSmallForCausalLM', 'BloomForCausalLM', 'CamembertForCausalLM', 'CodeGenForCausalLM', 'CpmAntForCausalLM', 'CTRLLMHeadModel', 'Data2VecTextForCausalLM', 'ElectraForCausalLM', 'ErnieForCausalLM', 'GitForCausalLM', 'GPT2LMHeadModel', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTNeoForCausalLM', 'GPTNeoXForCausalLM', 'GPTNeoXJapaneseForCausalLM', 'GPTJForCausalLM', 'LlamaForCausalLM', 'MarianForCausalLM', 'MBartForCausalLM', 'MegaForCausalLM', 'MegatronBertForCausalLM', 'MvpForCausalLM', 'OpenLlamaForCausalLM', 'OpenAIGPTLMHeadModel', 'OPTForCausalLM', 'PegasusForCausalLM', 'PLBartForCausalLM', 'ProphetNetForCausalLM', 'QDQBertLMHeadModel', 'ReformerModelWithLMHead', 'RemBertForCausalLM', 'RobertaForCausalLM', 'RobertaPreLayerNormForCausalLM', 'RoCBertForCausalLM', 'RoFormerForCausalLM', 'RwkvForCausalLM', 'Speech2Text2ForCausalLM', 'TransfoXLLMHeadModel', 'TrOCRForCausalLM', 'XGLMForCausalLM', 'XLMWithLMHeadModel', 'XLMProphetNetForCausalLM', 'XLMRobertaForCausalLM', 'XLMRobertaXLForCausalLM', 'XLNetLMHeadModel', 'XmodForCausalLM'].
willow-inference-server-wis-1 | huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
willow-inference-server-wis-1 | To disable this warning, you can either:
willow-inference-server-wis-1 | - Avoid using `tokenizers` before the fork if possible
willow-inference-server-wis-1 | - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
willow-inference-server-wis-1 | huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
willow-inference-server-wis-1 | To disable this warning, you can either:
willow-inference-server-wis-1 | - Avoid using `tokenizers` before the fork if possible
willow-inference-server-wis-1 | - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
willow-inference-server-wis-1 | huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
Anything I should try and fix? And what would I expect it to do if it is working? Tnx!
I found https://localhost:19000/chatbot/ :) Closing this as solved, tnx!
In both
main
andwisng
there's a version conflict inrequirements.txt
that triggers this error oncd chatbot && ./utilts.sh install 13B
:A workaround is to do this from within the container shell:
See also https://github.com/huggingface/transformers/issues/21128#issuecomment-1384031722