search

geronimi73 / 3090_shorts

minimal LLM scripts for 24GB VRAM GPUs. training, inference, whatever
28 stars 2 forks source link

run_id = f"OpenELM-450M_LR-{lr}_OA_{str(uuid.uuid4())}" #1

Closed elliotthwang closed 2 months ago

elliotthwang commented 2 months ago

Thank you your OPENELM ipynb tried a full finetune with HuggingFace SFTTrainer.

However, could you kind to advice how to set run_id = f"OpenELM-450M_LR-{lr}OA{str(uuid.uuid4())}" accordingly.

Regards

geronimi73 commented 2 months ago

run_id is just a string that uniquely identifies this run, it's only used for naming the output dir and the WANDB run. i have the bad habit of putting too much information in there.

these are the relevant lines where it is used:

training_arguments = TrainingArguments(
    output_dir = f"out_{run_id}",

...

wandb.init(
    project = "OpenELM", 
    name = run_id,
).log_code(include_fn=lambda path: path.endswith(".py") or path.endswith(".ipynb"))
elliotthwang commented 2 months ago

Thanks for your replying promptly!

Is it I could ignore the run_id setting?

geronimi73 commented 2 months ago

yes

elliotthwang commented 2 months ago

How about the bnb_config peft_config, should I put them into your finetune codes?

Thanks!

geronimi73 commented 2 months ago

How about the bnb_config peft_config, should I put them into your finetune codes?

None of those. The training code in the notebook is without quantization of the base model and no PEFT. It's a full finetune of a bfloat16 model

elliotthwang commented 2 months ago

Got!

Thanks