search

codefuse-ai / MFTCoder

High Accuracy and efficiency multi-task fine-tuning framework for Code LLMs. This work has been accepted by KDD 2024.
Other
631 stars 67 forks source link

模型微调完，合并时报错 NotImplementedError: Cannot copy out of meta tensor; no data! #45

Closed xxyp closed 9 months ago

xxyp commented 9 months ago

使用codefuse-deepseek-33B lora微调，执行ds_single_launch.sh 正常微调完后，合并模型报错

File /usr/local/lib/python3.8/dist-packages/safetensors/torch.py:404, in _tobytes(tensor, name) 396 raise ValueError( 397 f"You are trying to save a non contiguous tensor: {name} which is not allowed. It either means you" 398 " are trying to save tensors which are reference of each other in which case it's recommended to save" 399 " only the full tensors, and reslice at load time, or simply call .contiguous() on your tensor to" 400 " pack it before saving." 401 ) 402 if tensor.device.type != "cpu": 403 # Moving tensor to cpu before saving --> 404 tensor = tensor.to("cpu") 406 import ctypes 408 import numpy as np

NotImplementedError: Cannot copy out of meta tensor; no data!

chencyudel commented 9 months ago

@xxyp Post your script/command of merging. And did you merge on GPU?

xxyp commented 9 months ago

merge successful with single gpu