Open Herkaba opened 1 month ago
hi. it seems like the cuda drivers are not properly installed. i was able to use it jsut fine on 4090, 3090 and titan rtx. installing the cuda drivers properly will fix the issue.
you can also check if torch is detecting gpu.
import torch
if torch.cuda.is_available():
print("GPU is available")
print("GPU Name:", torch.cuda.get_device_name(0))
else:
print("GPU is not available")
I installed cuda toolkit and torch is detecting it but autotrain doesn't work either. Do i need to install cuda toolkit on my windows operating system or just in the wsl ubuntu ?
only WSL
also, make sure that correct path is available in CUDA_HOME
:
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
ohmm i changed the cuda home to /usr/local/cuda but i still get this error:
INFO | 2024-07-22 13:52:09 | autotrain.app.utils:kill_process_by_pid:52 - Sent SIGTERM to process with PID 3239
INFO | 2024-07-22 13:52:09 | autotrain.app.utils:get_running_jobs:26 - Killing PID: 3239
ERROR | 2024-07-22 13:52:05 | autotrain.trainers.common:wrapper:121 - No GPU found. A GPU is needed for quantization.
RuntimeError: No GPU found. A GPU is needed for quantization.
raise RuntimeError("No GPU found. A GPU is needed for quantization.")
File "/home/herkaba/.local/lib/python3.10/site-packages/transformers/quantizers/quantizer_bnb_4bit.py", line 62, in validate_environment
hf_quantizer.validate_environment(
File "/home/herkaba/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3279, in from_pretrained
return model_class.from_pretrained(
File "/home/herkaba/.local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
model = AutoModelForCausalLM.from_pretrained(
File "/home/herkaba/.local/lib/python3.10/site-packages/autotrain/trainers/clm/utils.py", line 649, in get_model
model = utils.get_model(config, tokenizer)
File "/home/herkaba/.local/lib/python3.10/site-packages/autotrain/trainers/clm/train_clm_sft.py", line 25, in train
train_sft(config)
File "/home/herkaba/.local/lib/python3.10/site-packages/autotrain/trainers/clm/main.py", line 28, in train
return func(*args, **kwargs)
File "/home/herkaba/.local/lib/python3.10/site-packages/autotrain/trainers/common.py", line 117, in wrapper
ERROR | 2024-07-22 13:52:05 | autotrain.trainers.common:wrapper:120 - train has failed due to an exception: Traceback (most recent call last):
INFO | 2024-07-22 13:52:05 | autotrain.trainers.clm.utils:get_model:635 - loading model...
INFO | 2024-07-22 13:52:05 | autotrain.trainers.clm.utils:get_model:627 - loading model config...
WARNING | 2024-07-22 13:52:05 | autotrain.trainers.clm.utils:get_model:625 - Unsloth not available, continuing without it...
INFO | 2024-07-22 13:52:05 | autotrain.trainers.clm.utils:get_model:583 - Can use unsloth: False
INFO | 2024-07-22 13:52:05 | autotrain.trainers.clm.utils:configure_block_size:548 - Using block size 1024
INFO | 2024-07-22 13:52:05 | autotrain.trainers.clm.utils:configure_training_args:485 - configuring training args
INFO | 2024-07-22 13:52:05 | autotrain.trainers.clm.utils:configure_logging_steps:480 - Logging steps: 4
INFO | 2024-07-22 13:52:05 | autotrain.trainers.clm.utils:configure_logging_steps:467 - configuring logging steps
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO | 2024-07-22 13:52:04 | autotrain.trainers.clm.utils:process_input_data:395 - Valid data: None
})
num_rows: 83
features: ['autotrain_text'],
INFO | 2024-07-22 13:52:04 | autotrain.trainers.clm.utils:process_input_data:394 - Train data: Dataset({
INFO | 2024-07-22 13:52:04 | autotrain.trainers.clm.utils:process_input_data:335 - loading dataset from disk
INFO | 2024-07-22 13:52:04 | autotrain.trainers.clm.train_clm_sft:train:12 - Starting SFT training...
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
INFO | 2024-07-22 13:52:00 | autotrain.backends.local:create:13 - Training PID: 3239
This issue is stale because it has been open for 30 days with no activity.
I ran on huggingface's hub to run through the process without using the GPU and had the same problem. The same problem occurs when I start training after the dataset is downloaded.
Prerequisites
Backend
Local
Interface Used
UI
CLI Command
No response
UI Screenshots & Parameters
Error Logs
Device 0: NVIDIA GeForce RTX 4060 - 1387MiB/8188MiB
INFO | 2024-07-21 12:47:42 | autotrain.app.utils:kill_process_by_pid:52 - Sent SIGTERM to process with PID 4438
INFO | 2024-07-21 12:47:42 | autotrain.app.utils:get_running_jobs:26 - Killing PID: 4438
ERROR | 2024-07-21 12:47:38 | autotrain.trainers.common:wrapper:121 - No GPU found. A GPU is needed for quantization.
RuntimeError: No GPU found. A GPU is needed for quantization.
raise RuntimeError("No GPU found. A GPU is needed for quantization.")
File "/home/herkaba/.local/lib/python3.10/site-packages/transformers/quantizers/quantizer_bnb_4bit.py", line 62, in validate_environment
hf_quantizer.validate_environment(
File "/home/herkaba/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3279, in from_pretrained
return model_class.from_pretrained(
File "/home/herkaba/.local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
model = AutoModelForCausalLM.from_pretrained(
File "/home/herkaba/.local/lib/python3.10/site-packages/autotrain/trainers/clm/utils.py", line 649, in get_model
model = utils.get_model(config, tokenizer)
File "/home/herkaba/.local/lib/python3.10/site-packages/autotrain/trainers/clm/train_clm_sft.py", line 25, in train
train_sft(config)
File "/home/herkaba/.local/lib/python3.10/site-packages/autotrain/trainers/clm/main.py", line 28, in train
return func(*args, **kwargs)
File "/home/herkaba/.local/lib/python3.10/site-packages/autotrain/trainers/common.py", line 117, in wrapper
ERROR | 2024-07-21 12:47:38 | autotrain.trainers.common:wrapper:120 - train has failed due to an exception: Traceback (most recent call last):
INFO | 2024-07-21 12:47:37 | autotrain.trainers.clm.utils:get_model:635 - loading model...
INFO | 2024-07-21 12:47:37 | autotrain.trainers.clm.utils:get_model:627 - loading model config...
WARNING | 2024-07-21 12:47:37 | autotrain.trainers.clm.utils:get_model:625 - Unsloth not available, continuing without it...
INFO | 2024-07-21 12:47:37 | autotrain.trainers.clm.utils:get_model:583 - Can use unsloth: False
INFO | 2024-07-21 12:47:37 | autotrain.trainers.clm.utils:configure_block_size:548 - Using block size 1024
INFO | 2024-07-21 12:47:37 | autotrain.trainers.clm.utils:configure_training_args:485 - configuring training args
INFO | 2024-07-21 12:47:37 | autotrain.trainers.clm.utils:configure_logging_steps:480 - Logging steps: 2
INFO | 2024-07-21 12:47:37 | autotrain.trainers.clm.utils:configure_logging_steps:467 - configuring logging steps
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO | 2024-07-21 12:47:37 | autotrain.trainers.clm.utils:process_input_data:395 - Valid data: None
})
num_rows: 83
features: ['autotrain_text'],
INFO | 2024-07-21 12:47:37 | autotrain.trainers.clm.utils:process_input_data:394 - Train data: Dataset({
INFO | 2024-07-21 12:47:37 | autotrain.trainers.clm.utils:process_input_data:335 - loading dataset from disk
INFO | 2024-07-21 12:47:37 | autotrain.trainers.clm.train_clm_sft:train:12 - Starting SFT training...
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
INFO | 2024-07-21 12:47:32 | autotrain.backends.local:create:13 - Training PID: 4438
INFO | 2024-07-21 12:47:32 | autotrain.commands:launch_command:401 - {'model': 'meta-llama/Meta-Llama-3-8B-Instruct', 'project_name': 'trainTeszt', 'data_path': 'trainTeszt/autotrain-data', 'train_split': 'train', 'valid_split': None, 'add_eos_token': True, 'block_size': 1024, 'model_max_length': 2048, 'padding': 'right', 'trainer': 'sft', 'use_flash_attention_2': False, 'log': 'tensorboard', 'disable_gradient_checkpointing': False, 'logging_steps': -1, 'eval_strategy': 'epoch', 'save_total_limit': 1, 'auto_find_batch_size': False, 'mixed_precision': 'fp16', 'lr': 3e-05, 'epochs': 3, 'batch_size': 6, 'warmup_ratio': 0.1, 'gradient_accumulation': 4, 'optimizer': 'adamw_torch', 'scheduler': 'linear', 'weight_decay': 0.0, 'max_grad_norm': 1.0, 'seed': 42, 'chat_template': 'none', 'quantization': 'int4', 'target_modules': 'all-linear', 'merge_adapter': False, 'peft': True, 'lora_r': 16, 'lora_alpha': 32, 'lora_dropout': 0.05, 'model_ref': None, 'dpo_beta': 0.1, 'max_prompt_length': 128, 'max_completion_length': None, 'prompt_text_column': 'autotrain_prompt', 'text_column': 'autotrain_text', 'rejected_text_column': 'autotrain_rejected_text', 'push_to_hub': True, 'username': 'herkaba', 'token': '*****', 'unsloth': False}
INFO | 2024-07-21 12:47:32 | autotrain.commands:launch_command:400 - ['accelerate', 'launch', '--num_machines', '1', '--num_processes', '1', '--mixed_precision', 'fp16', '-m', 'autotrain.trainers.clm', '--training_config', 'trainTeszt/training_params.json']
INFO | 2024-07-21 12:47:32 | autotrain.backends.local:create:8 - Starting local training...
Saving the dataset (1/1 shards): 100%|ââââââââââ| 83/83 [00:00<00:00, 68515.50 examples/s]
Saving the dataset (1/1 shards): 100%|ââââââââââ| 83/83 [00:00<00:00, 74290.92 examples/s]
Saving the dataset (0/1 shards): 0%| | 0/83 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|ââââââââââ| 83/83 [00:00<00:00, 20154.42 examples/s]
Saving the dataset (1/1 shards): 100%|ââââââââââ| 83/83 [00:00<00:00, 20723.09 examples/s]
Saving the dataset (0/1 shards): 0%| | 0/83 [00:00<?, ? examples/s]
INFO | 2024-07-21 12:47:32 | autotrain.app.ui_routes:handle_form:608 - Column mapping: {'text': 'text'}
INFO | 2024-07-21 12:47:32 | autotrain.app.ui_routes:handle_form:607 - Task: lm_training
INFO | 2024-07-21 12:47:32 | autotrain.app.ui_routes:handle_form:491 - hardware: local-ui
INFO | 2024-07-21 12:44:09 | autotrain.app.ui_routes:fetch_params:376 - Task: llm:sft
INFO | 2024-07-21 12:44:04 | autotrain.app.ui_routes:fetch_params:376 - Task: llm:sft
INFO | 2024-07-21 12:44:01 | autotrain.app.ui_routes:fetch_params:376 - Task: llm:reward
INFO | 2024-07-21 12:43:56 | autotrain.app.ui_routes:fetch_params:376 - Task: llm:dpo
INFO | 2024-07-21 12:43:54 | autotrain.app.ui_routes:fetch_params:376 - Task: llm:generic
INFO | 2024-07-21 12:43:51 | autotrain.app.ui_routes:fetch_params:376 - Task: llm:orpo
INFO | 2024-07-21 12:43:47 | autotrain.app.ui_routes:fetch_params:376 - Task: llm:sft
INFO | 2024-07-21 12:43:45 | autotrain.app.ui_routes:fetch_params:376 - Task: llm:generic
INFO | 2024-07-21 12:43:28 | autotrain.app.ui_routes:fetch_params:376 - Task: llm:sft
INFO: 127.0.0.1:45810 - "GET / HTTP/1.1" 307 Temporary Redirect
INFO: Uvicorn running on http://127.0.0.1:7860 (Press CTRL+C to quit)
INFO: Application startup complete.
INFO: Waiting for application startup.
INFO: Started server process [3135]
INFO | 2024-07-21 12:43:25 | autotrain.app.app::24 - AutoTrain started successfully
INFO | 2024-07-21 12:43:25 | autotrain.app.app::23 - AutoTrain version: 0.8.4
INFO | 2024-07-21 12:43:25 | autotrain.app.app::13 - Starting AutoTrain...
INFO | 2024-07-21 12:43:25 | autotrain.app.ui_routes::293 - AutoTrain started successfully
INFO | 2024-07-21 12:43:20 | autotrain.app.ui_routes::31 - Starting AutoTrain...
Additional Information
I have cuda toolkit installed and i am using a wsl ubuntu on windows 11