search

Lightning-AI / litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
https://lightning.ai
Apache License 2.0
9.76k stars 974 forks source link

Trainer args consistency #951

Closed rasbt closed 6 months ago

rasbt commented 6 months ago

For consistency, should it be io.destination_path (or perhaps even better io.out_dir) etc. in 

python scripts/prepare_alpaca.py \
   --destination_path data/alpaca \
   --checkpoint_dir checkpoints/google/gemma-2b

python finetune/lora.py \
   --io.train_data_dir data/alpaca \
   --io.checkpoint_dir checkpoints/google/gemma-2b
carmocca commented 6 months ago

Maybe.

This can be done in one of two ways:

  1. Add all those new arguments to IOArgs, the downside is that you'll need to update the argument validation for all scripts.
  2. Create a new PrepareArgs and then do io: PrepareArgs = PrepareArgs() in the prepare_*.py scripts. From the command line it's the same but internally it's using a different class.

I think it depends on how many arguments are shared across scripts. We should also consider how it will look after #950

carmocca commented 6 months ago

@awaelchli Is refactoring the current prepare_*.py scripts in scope for #950?

awaelchli commented 6 months ago

Yes, with https://github.com/Lightning-AI/lit-gpt/pull/950 the goal is to replace these prepare scripts (with the exception of the pretraining datasets which need to be optimized in advance). We haven't made any decisions yet and I'd like to discuss it, but my vision was that the scripts get a datamodule as input which encapsulate all data related args.

carmocca commented 6 months ago

Cool. In that case https://github.com/Lightning-AI/lit-gpt/issues/954 replaces this issue