Failed to run - required architechture?

[andrewjaykeller]

Hi! I love this project, thanks for all of your foundational work.

I am trying to run on an M1 mac, and am getting this error:

/workspaces/vscode-remote-try-python/scripts/../src/train_gpt.py:40: DeprecationWarning: 
Pyarrow will become a required dependency of pandas in the next major release of pandas (pandas 3.0),
(to allow more performant data types, such as the Arrow string type, and better interoperability with other libraries)
but was not found to be installed on your system.
If this would cause problems for you,
please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466

  import pandas as pd
/home/vscode/.local/lib/python3.11/site-packages/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
Number of subjects loaded:  0
Number of subjects loaded:  0
Traceback (most recent call last):
  File "/workspaces/vscode-remote-try-python/scripts/../src/train_gpt.py", line 997, in <module>
    trainer = train()
              ^^^^^^^
  File "/workspaces/vscode-remote-try-python/scripts/../src/train_gpt.py", line 192, in train
    trainer = make_trainer(
              ^^^^^^^^^^^^^
  File "/workspaces/vscode-remote-try-python/src/trainer/make.py", line 164, in make_trainer
    trainer_args = TrainingArguments(
                   ^^^^^^^^^^^^^^^^^^
  File "<string>", line 123, in __init__
  File "/home/vscode/.local/lib/python3.11/site-packages/transformers/training_args.py", line 1534, in __post_init__
    raise ValueError(
ValueError: FP16 Mixed precision training with AMP or APEX (`--fp16`) and FP16 half precision evaluation (`--fp16_full_eval`) can only be used on CUDA or NPU devices or certain XPU devices (with IPEX).

Should I change over to an NVIDIA device that runs CUDA? How do I point the project to another location to read the files? I saw a file called train_config.json but don't see an example in the repo.

From what I can gather, it's at least:

{
    "log_dir": "/path/to/log/directory"
}

Thanks!

AJ

[wenhui0206]

Hi AJ, Thanks for your interest in this project. Yes, there are dependencies on CUDA. And it is a large model, best to be trained on a GPU with at least 12GB memory. I updated the train.sh and finetune.sh files, now you can specify the args 'train-data-path' to your own data path. 'log_dir' refers to the output directory during training.

[andrewjaykeller]

Thanks! I'll close this!