chatglm-6b chase fine-tuning error on gen_train_eval_data

[baiclamp]

(dbgpt_hub) aistudio@jupyter-4913610-7006586:~/DB_GPT_Hub/DB-GPT-Hub-0.2.0$ sh dbgpt_hub/scripts/gen_train_eval_data.sh

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin /home/aistudio/.conda/envs/dbgpt_hub/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda116_nocublaslt.so /home/aistudio/.conda/envs/dbgpt_hub/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /home/aistudio/.conda/envs/dbgpt_hub did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) /home/aistudio/.conda/envs/dbgpt_hub/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda-11.2/targets/x86_64-linux'), PosixPath('/usr/local/cuda-11.2/lib64'), PosixPath('/usr/local/cuda-11.2/targets/x86_64-linux/lib')} warn(msg) CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0 CUDA SETUP: Highest compute capability among GPUs detected: 7.0 CUDA SETUP: Detected CUDA version 116 /home/aistudio/.conda/envs/dbgpt_hub/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU! warn(msg) CUDA SETUP: Loading binary /home/aistudio/.conda/envs/dbgpt_hub/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda116_nocublaslt.so... [2023-11-08 10:06:45,322] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect) 0%| | 0/3949 [00:00<?, ?it/s] Traceback (most recent call last): File "/home/aistudio/DB_GPT_Hub/DB-GPT-Hub-0.2.0/dbgpt_hub/data_process/sql_data_process.py", line 99, in precess.create_sft_raw_data() File "/home/aistudio/DB_GPT_Hub/DB-GPT-Hub-0.2.0/dbgpt_hub/data_process/sql_data_process.py", line 83, in create_sft_raw_data self.decode_json_file( File "/home/aistudio/DB_GPT_Hub/DB-GPT-Hub-0.2.0/dbgpt_hub/data_process/sql_data_process.py", line 65, in decode_json_file if data["db_id"] in db_dict.keys(): KeyError: 'db_id'

MODELS_PARENT_PATH = "/home/aistudio/DB_GPT_Hub/DB-GPT-Hub-0.2.0/model/" DEFAULT_FT_MODEL_NAME = "chatglm2-6b" MODEL_PATH = os.path.join(MODELS_PARENT_PATH, DEFAULT_FT_MODEL_NAME)

MODEL_PATH = os.path.join(ROOT_PATH, "model")

ADAPTER_PATH = os.path.join(ROOT_PATH, "dbgpt_hub/output/adapter") MERGED_MODELS = os.path.join(ROOT_PATH, "dbgpt_hub/output/merged_models")

DATA_PATH = "/root/autodl-tmp/data/spider/pre_processed_data"

OUT_DIR= "/root/autodl-tmp/codellama"

DATA_PATH = os.path.join(ROOT_PATH, "dbgpt_hub/data") PREDICTED_DATA_PATH = os.path.join(ROOT_PATH, "dbgpt_hub/data/eval_data/dev_sql.json") PREDICTED_OUT_FILENAME = "pred_sql.sql"

OUT_DIR = os.path.join(DATA_PATH, "out_pred")

OUT_DIR = os.path.join(ROOT_PATH, "dbgpt_hub/output/")

model constants

IGNORE_INDEX = -100 DEFAULT_PAD_TOKEN = "[PAD]" DEFAULT_EOS_TOKEN = "" DEFAULT_BOS_TOKEN = "" DEFAULT_UNK_TOKEN = ""

LOG_FILE_NAME = "trainer_log.jsonl"

head_state_dict,model save name

VALUE_HEAD_FILE_NAME = "value_head.bin"

output ,finetuning_args save_to_json name

FINETUNING_ARGS_NAME = "finetuning_args.json"

when prepare_model_for_training ,layer_norm_names

LAYERNORM_NAMES = ["norm", "ln_f", "ln_attn", "ln_mlp"] EXT2TYPE = {"csv": "csv", "json": "json", "jsonl": "json", "txt": "text"}

text2sql dataset information for processing sql data

SQL_DATA_INFO = [ { "data_source": "chase", "train_file": ["chase_train.json"], "dev_file": ["chase_dev.json"], "tables_file": "chase_tables.json", "db_id_name": "db_id", "is_multiple_turn": False

}

[wangzaistone]

What's your code commit_id ? It seems is the data formate change problem, lately ,there is a contributor add some new dataset.