danielfleischer
commented
1 month ago Closed chenGIEGIE closed 1 week ago
danielfleischer
commented
1 month ago danielfleischer
commented
1 month ago 
chenGIEGIE
commented
3 weeks ago Thank you very much. I changed the model_name_or_path using the method you provided, but now I encounter a new problem. I am using the large model internlm2_5-20b-chat, and the system shows that the model I am using is missing the module '''qkv_proj'''. How should I modify the code? `(RAGgraphcityrun) root@intern-studio-50166964:~/RAGFoundry# python training.py -cp configs/paper -cn training-asqa data_file=asqa-raft-cot-train.jsonl output_key=generated_answer train.output_dir=./trained_models_cot/ [2024-08-17 00:02:27,377][main][INFO] - model: target: ragfoundry.models.hf.HFTrain model_name_or_path: internlm/internlm2_5-20b-chat load_in_4bit: false load_in_8bit: true torch_dtype: null device_map: null trust_remote_code: true lora: bias: none fan_in_fan_out: false layers_pattern: null layers_to_transform: null lora_alpha: 16 lora_dropout: 0.1 peft_type: LORA r: 16 target_modules:
[2024-08-17 00:02:27,377][main][INFO] - Experiment name: None [2024-08-17 00:02:27,377][main][INFO] - Output path: ./trained_models_cot/ [2024-08-17 00:02:27,377][main][INFO] - Loading dataset: asqa-raft-cot-train.jsonl [2024-08-17 00:02:28,385][main][INFO] - Loading instruction from file ragfoundry/processing/prompts/prompt_instructions/qa.txt... [2024-08-17 00:02:28,387][main][INFO] - Loaded instruction: You are a helpful question answerer who can provide an answer given a question and relevant context. The repository for internlm/internlm2_5-20b-chat contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/internlm/internlm2_5-20b-chat. You can avoid this prompt in future by passing the argument trust_remote_code=True.
Do you wish to run the custom code? [y/N] y /root/.conda/envs/RAGgraphcityrun/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True. warnings.warn( The load_in_4bit and load_in_8bit arguments are deprecated and will be removed in the future versions. Please, pass a BitsAndBytesConfig object in quantization_config argument instead. low_cpu_mem_usage was None, now set to True since model is quantized. Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 21/21 [05:31<00:00, 15.80s/it] generation_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 123/123 [00:00<00:00, 896kB/s] [2024-08-17 00:08:13,486][ragfoundry.models.hf][INFO] - Loaded model: InternLM2ForCausalLM( (model): InternLM2Model( (tok_embeddings): Embedding(92544, 6144, padding_idx=2) (layers): ModuleList( (0-47): 48 x InternLM2DecoderLayer( (attention): InternLM2Attention( (wqkv): Linear8bitLt(in_features=6144, out_features=8192, bias=False) (wo): Linear8bitLt(in_features=6144, out_features=6144, bias=False) (rotary_emb): InternLM2DynamicNTKScalingRotaryEmbedding() ) (feed_forward): InternLM2MLP( (w1): Linear8bitLt(in_features=6144, out_features=16384, bias=False) (w3): Linear8bitLt(in_features=6144, out_features=16384, bias=False) (w2): Linear8bitLt(in_features=16384, out_features=6144, bias=False) (act_fn): SiLU() ) (attention_norm): InternLM2RMSNorm() (ffn_norm): InternLM2RMSNorm() ) ) (norm): InternLM2RMSNorm() ) (output): Linear(in_features=6144, out_features=92544, bias=False) ) [2024-08-17 00:08:13,487][ragfoundry.models.hf][INFO] - Initializing LORA based on {'bias': 'none', 'fan_in_fan_out': False, 'layers_pattern': None, 'layers_to_transform': None, 'lora_alpha': 16, 'lora_dropout': 0.1, 'peft_type': 'LORA', 'r': 16, 'target_modules': ['qkv_proj'], 'task_type': 'CAUSAL_LM', 'use_rslora': True} Error executing job with overrides: ['data_file=asqa-raft-cot-train.jsonl', 'output_key=generated_answer', 'train.output_dir=./trained_models_cot/'] Traceback (most recent call last): File "/root/.conda/envs/RAGgraphcityrun/lib/python3.10/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target return target(*args, kwargs) File "/root/RAGFoundry/ragfoundry/models/hf.py", line 151, in init self.model = get_peft_model(self.model, LoraConfig(lora)) File "/root/.conda/envs/RAGgraphcityrun/lib/python3.10/site-packages/peft/mapping.py", line 149, in get_peft_model return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config, adapter_name=adapter_name) File "/root/.conda/envs/RAGgraphcityrun/lib/python3.10/site-packages/peft/peft_model.py", line 1395, in init super().init(model, peft_config, adapter_name) File "/root/.conda/envs/RAGgraphcityrun/lib/python3.10/site-packages/peft/peft_model.py", line 138, in init self.base_model = cls(model, {adapter_name: peft_config}, adapter_name) File "/root/.conda/envs/RAGgraphcityrun/lib/python3.10/site-packages/peft/tuners/lora/model.py", line 139, in init super().init(model, config, adapter_name) File "/root/.conda/envs/RAGgraphcityrun/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 166, in init self.inject_adapter(self.model, adapter_name) File "/root/.conda/envs/RAGgraphcityrun/lib/python3.10/site-packages/peft/tuners/tuners_utils.py", line 375, in inject_adapter raise ValueError( ValueError: Target modules {'qkv_proj'} not found in the base model. Please check the target modules and try again.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/RAGFoundry/training.py", line 123, in
danielfleischer
commented
3 weeks ago I'm not familiar with your LLM but in general the modules you use with LORA in target_modules should exist in the LLM. qkv_proj is an example, a module from Phi-3; you need to adjust this for your own LLM.
chenGIEGIE
commented
3 weeks ago thanks，I checked the data from previous data enhancement today and found that the data in my positive key is empty. I did not make many changes to the retrieval code. Is this normal? If it is wrong, where might the problem be?(I just changed the URL in the qrant.yaml file to memory.)
`components: retriever: init_parameters: document_store: type: haystack_integrations.document_stores.qdrant.document_store.QdrantDocumentStore init_parameters: location: ":memory:" index: wikipedia embedding_dim: 768 similarity: dot_product write_batch_size: 50 top_k: 5 type: haystack_integrations.components.retrievers.qdrant.retriever.QdrantEmbeddingRetriever text_embedder: init_parameters: batch_size: 64 model: BAAI/llm-embedder prefix: "Represent this query for retrieving relevant documents: " device: type: haystack.components.embedders.sentence_transformers_text_embedder.SentenceTransformersTextEmbedder connections:
The results show that the dataset I generated does not have a relevant index under the positive key: {"ambiguous_question":"When does the new bunk'd come out?","qa_pairs":[{"context":"No context provided","question":"When does episode 42 of bunk'd come out?","short_answers":["May 24, 2017"],"wikipage":null},{"context":"No context provided","question":"When does episode 41 of bunk'd come out?","short_answers":["April 28, 2017"],"wikipage":null},{"context":"No context provided","question":"When does episode 40 of bunk'd come out?","short_answers":["April 21, 2017"],"wikipage":null}],"wikipages":[{"title":"List of Bunk'd episodes","url":"https:\/\/en.wikipedia.org\/wiki\/List%20of%20Bunk%27d%20episodes"}],"annotations":[{"knowledge":[{"content":null,"wikipage":"List of Bunk'd episodes"}],"long_answer":"The new bunk'd episode 41 comes out on April 21, 2017, episode 42 comes out on April 28, 2017 and episode 42 is due to come out on May 24, 2017. "}],"sample_id":"-5742327688291876861","answer-long":["The new bunk'd episode 41 comes out on April 21, 2017, episode 42 comes out on April 28, 2017 and episode 42 is due to come out on May 24, 2017. "],"answer-short":[["may 24 2017"],["april 28 2017"],["april 21 2017"]],"answers":["The new bunk'd episode 41 comes out on April 21, 2017, episode 42 comes out on April 28, 2017 and episode 42 is due to come out on May 24, 2017. "],"query":"When does the new bunk'd come out?","positive_passages":[],"negative_passages":[]}
danielfleischer
commented
3 weeks ago Are there any documents in your corpus or the Qdrant index empty, perhaps? see issue #4 and #6.
What models does this library support? Is there any way to customize the models? Thanks