Closed amwork2020 closed 11 months ago
Jintao-Huang
commented
11 months ago 你是不是将train_dataset_sample设置成了0. 我看到你的输出中打印的数据集内容是：
[INFO:swift] train_dataset: Dataset({ features: ['system', 'query', 'response', 'history'], num_rows: 0 }) [INFO:swift] val_dataset: Dataset({ features: ['system', 'query', 'response', 'history'], num_rows: 152 })
amwork2020
commented
11 months ago train_dataset_sample
amwork2020
commented
11 months ago 你是不是将train_dataset_sample设置成了0. 我看到你的输出中打印的数据集内容是：
[INFO:swift] train_dataset: Dataset({ features: ['system', 'query', 'response', 'history'], num_rows: 0 }) [INFO:swift] val_dataset: Dataset({ features: ['system', 'query', 'response', 'history'], num_rows: 152 }) 代码注释好了
Jintao-Huang
commented
11 months ago 100%|███████████████████████████████████████████████████████████████████████████| 598185/598185 [00:22<00:00, 27069.23it/s] 100%|████████████████████████████████████████████████████████████████████████████| 39964/39964 [00:00<00:00, 624573.51it/s] 100%|█████████████████████████████████████████████████████████████████████████████████| 360/360 [00:00<00:00, 11020.08it/s] 100%|████████████████████████████████████████████████████████████████████████████████| 152/152 [00:00<00:00, 720377.64it/s] [INFO:swift] train_dataset: Dataset({ features: ['system', 'query', 'response', 'history'], num_rows: 39964 }) [INFO:swift] val_dataset: Dataset({ features: ['system', 'query', 'response', 'history'], num_rows: 152 })
这是我这里的输出，你检查一下数据集是否完整，可以删一下重新下载一下试试哈
amwork2020
commented
11 months ago 100%|███████████████████████████████████████████████████████████████████████████| 598185/598185 [00:22<00:00, 27069.23it/s] 100%|████████████████████████████████████████████████████████████████████████████| 39964/39964 [00:00<00:00, 624573.51it/s] 100%|█████████████████████████████████████████████████████████████████████████████████| 360/360 [00:00<00:00, 11020.08it/s] 100%|████████████████████████████████████████████████████████████████████████████████| 152/152 [00:00<00:00, 720377.64it/s] [INFO:swift] train_dataset: Dataset({ features: ['system', 'query', 'response', 'history'], num_rows: 39964 }) [INFO:swift] val_dataset: Dataset({ features: ['system', 'query', 'response', 'history'], num_rows: 152 })
这是我这里的输出，你检查一下数据集是否完整，可以删一下重新下载一下试试哈
确实是，删除一下就好了，谢谢！
Jintao-Huang
commented
11 months ago 好嘞
INFO:modelscope] No subset_name specified, defaulting to the default [WARNING:modelscope] Reusing dataset ms_agent-bench (/u01/liuys/.cache/modelscope/hub/datasets/damo/MSAgent-Bench/master/data_files) [INFO:modelscope] Generating dataset ms_agent-bench (/u01/liuys/.cache/modelscope/hub/datasets/damo/MSAgent-Bench/master/data_files) [INFO:modelscope] Reusing cached meta-data file: /u01/liuys/.cache/modelscope/hub/datasets/damo/MSAgent-Bench/master/data_files/d7bde86fe25cc1973db0962e0dfe0b07 [INFO:modelscope] Reusing cached meta-data file: /u01/liuys/.cache/modelscope/hub/datasets/damo/MSAgent-Bench/master/data_files/466e047df4e2dc8c730e0ba6e1d60e41 Downloading data files: 0it [00:00, ?it/s] Extracting data files: 0it [00:00, ?it/s] 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 153600/153600 [00:07<00:00, 19888.54it/s] 0it [00:00, ?it/s] 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 360/360 [00:00<00:00, 7210.39it/s] 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 152/152 [00:00<00:00, 577476.64it/s] [INFO:swift] train_dataset: Dataset({ features: ['system', 'query', 'response', 'history'], num_rows: 0 }) [INFO:swift] val_dataset: Dataset({ features: ['system', 'query', 'response', 'history'], num_rows: 152 }) 0it [00:00, ?it/s] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 152/152 [00:00<00:00, 500.85it/s] /u01/liuys/swift/package/swift/utils/llm_utils.py:25: RuntimeWarning: Mean of empty slice. mean = _token_len.mean().item() /u01/liuys/anaconda3/envs/ms-sft/lib/python3.10/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide ret = ret.dtype.type(ret / rcount) /u01/liuys/anaconda3/envs/ms-sft/lib/python3.10/site-packages/numpy/core/_methods.py:206: RuntimeWarning: Degrees of freedom <= 0 for slice ret = _var(a, axis=axis, dtype=dtype, out=out, ddof=ddof, /u01/liuys/anaconda3/envs/ms-sft/lib/python3.10/site-packages/numpy/core/_methods.py:163: RuntimeWarning: invalid value encountered in divide arrmean = um.true_divide(arrmean, div, out=arrmean, /u01/liuys/anaconda3/envs/ms-sft/lib/python3.10/site-packages/numpy/core/_methods.py:198: RuntimeWarning: invalid value encountered in scalar divide ret = ret.dtype.type(ret / rcount) Traceback (most recent call last): File "/u01/liuys/swift/examples/pytorch/llm/src/llm_sft.py", line 241, in
llm_sft(args)
File "/u01/liuys/swift/examples/pytorch/llm/src/llm_sft.py", line 110, in llm_sft
stat_dataset(train_dataset)
File "/u01/liuys/swift/package/swift/utils/llm_utils.py", line 27, in statdataset
min = _token_len.min().item()
File "/u01/liuys/anaconda3/envs/ms-sft/lib/python3.10/site-packages/numpy/core/_methods.py", line 45, in _amin
return umr_minimum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation minimum which has no identity