Open MultiLLM1 opened 11 months ago
Lora微调时放开了更多层(0-4,10-14),但是加载的时候好像加载不到其他层的参数,但是找不到哪里可以改,用AutoModel和FineTuneVisualGLMModel的方法都不行,请问加载的时候在哪里改加载其他层的微调参数呢?报错信息如下:
ValueError: Missing keys for inference: ['transformer.layers.1.attention.query_key_value.weight', 'transformer.layers.1.attention.query_key_value.bias', 'transformer.layers.1.attention.dense.weight', 'transformer.layers.1.attention.dense.bias', 'transformer.layers.2.attention.query_key_value.weight', 'transformer.layers.2.attention.query_key_value.bias', 'transformer.layers.2.attention.dense.weight', 'transformer.layers.2.attention.dense.bias', 'transformer.layers.3.attention.query_key_value.weight', 'transformer.layers.3.attention.query_key_value.bias', 'transformer.layers.3.attention.dense.weight', 'transformer.layers.3.attention.dense.bias', 'transformer.layers.4.attention.query_key_value.weight', 'transformer.layers.4.attention.query_key_value.bias', 'transformer.layers.4.attention.dense.weight', 'transformer.layers.4.attention.dense.bias', 'transformer.layers.10.attention.query_key_value.weight', 'transformer.layers.10.attention.query_key_value.bias', 'transformer.layers.10.attention.dense.weight', 'transformer.layers.10.attention.dense.bias', 'transformer.layers.11.attention.query_key_value.weight', 'transformer.layers.11.attention.query_key_value.bias', 'transformer.layers.11.attention.dense.weight', 'transformer.layers.11.attention.dense.bias', 'transformer.layers.12.attention.query_key_value.weight', 'transformer.layers.12.attention.query_key_value.bias', 'transformer.layers.12.attention.dense.weight', 'transformer.layers.12.attention.dense.bias', 'transformer.layers.13.attention.query_key_value.weight', 'transformer.layers.13.attention.query_key_value.bias', 'transformer.layers.13.attention.dense.weight', 'transformer.layers.13.attention.dense.bias']. If you still want to inference anyway, pass --force_inference to args.
看起来只找到了0、14层的,其他层要怎么指定
Lora微调时放开了更多层(0-4,10-14),但是加载的时候好像加载不到其他层的参数,但是找不到哪里可以改,用AutoModel和FineTuneVisualGLMModel的方法都不行,请问加载的时候在哪里改加载其他层的微调参数呢?报错信息如下:
看起来只找到了0、14层的,其他层要怎么指定