search

deepseek-ai / DeepSeek-Coder

DeepSeek Coder: Let the Code Write Itself
https://coder.deepseek.com/
MIT License
6.82k stars 472 forks source link

I try Fine-tune DeepSeek-Coder #181

Closed Siwakonrome closed 1 month ago

Siwakonrome commented 1 month ago

Error with incorrect path and How using my own dataset from csv to Fine-tune model

Load dataset

from datasets import load_dataset dataset = load_dataset("nickrosh/Evol-Instruct-Code-80k-v1", cache_dir="/home/siwakon/FIBO/DeepSeek-Coder/finetune/datasets_input")

/home/siwakon/FIBO/DeepSeek-Coder/finetune/datasets_input

My cmd

deepspeed finetune_deepseekcoder.py \ --model_name_or_path deepseek-ai/deepseek-coder-1.3b-instruct \ --data_path /home/siwakon/FIBO/DeepSeek-Coder/finetune/datasets_input/nickrosh___evol-instruct-code-80k-v1 \ --output_dir /home/siwakon/FIBO/DeepSeek-Coder/finetune/datasets_output \ --num_train_epochs 3 \ --model_max_length 1024 \ --per_device_train_batch_size 16 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 4 \ --evaluation_strategy "no" \ --save_strategy "steps" \ --save_steps 100 \ --save_total_limit 100 \ --learning_rate 2e-5 \ --warmup_steps 10 \ --logging_steps 1 \ --lr_scheduler_type "cosine" \ --gradient_checkpointing True \ --report_to "tensorboard" \ --deepspeed configs/ds_config_zero3.json \ --bf16 False

Raise error

FileNotFoundError: Unable to find '/home/siwakon/FIBO/DeepSeek-Coder/finetune/datasets_input' [2024-09-16 14:17:32,879] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 30137 [2024-09-16 14:17:32,880] [ERROR] [launch.py:321:sigkill_handler] ['/usr/bin/python3', '-u', 'finetune_deepseekcoder.py', '--local_rank=0', '--model_name_or_path', 'deepseek-ai/deepseek-coder-1.3b-instruct', '--data_path', '/home/siwakon/FIBO/DeepSeek-Coder/finetune/datasets_input/nickrosh___evol-instruct-code-80k-v1', '--output_dir', '/home/siwakon/FIBO/DeepSeek-Coder/finetune/datasets_output', '--num_train_epochs', '3', '--model_max_length', '1024', '--per_device_train_batch_size', '16', '--per_device_eval_batch_size', '1', '--gradient_accumulation_steps', '4', '--evaluation_strategy', 'no', '--save_strategy', 'steps', '--save_steps', '100', '--save_total_limit', '100', '--learning_rate', '2e-5', '--warmup_steps', '10', '--logging_steps', '1', '--lr_scheduler_type', 'cosine', '--gradient_checkpointing', 'True', '--report_to', 'tensorboard', '--deepspeed', 'configs/ds_config_zero3.json', '--bf16', 'False'] exits with return code = 1

nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Thu_Nov_18_09:45:30_PST_2021 Cuda compilation tools, release 11.5, V11.5.119 Build cuda_11.5.r11.5/compiler.30672275_0

NVIDIA GeForce GTX 1660 Ti (Driver Version: 550.107.02 CUDA Version: 12.4)

Thank you.

soloice commented 1 month ago

This should be irrelevant to deepseek coder. Just check your data path.