Open SalmanMohammadi opened 3 months ago
@SalmanMohammadi , do you have some example of what you would like to change?
Good question. This is the current output:
(venv) combuter:torchtune salmanmohammadi$ tune ls
W0805 18:37:08.061000 8424225472 torch/distributed/elastic/multiprocessing/redirects.py:29] NOTE: Redirects are currently not supported in Windows or MacOs.
RECIPE CONFIG
full_finetune_single_device llama2/7B_full_low_memory
code_llama2/7B_full_low_memory
llama3/8B_full_single_device
llama3_1/8B_full_single_device
mistral/7B_full_low_memory
phi3/mini_full_low_memory
qwen2/7B_full_low_memory
full_finetune_distributed llama2/7B_full
llama2/13B_full
llama3/8B_full
llama3_1/8B_full
llama3/70B_full
llama3_1/70B_full
mistral/7B_full
gemma/2B_full
gemma/7B_full
phi3/mini_full
qwen2/7B_full
lora_finetune_single_device llama2/7B_lora_single_device
llama2/7B_qlora_single_device
code_llama2/7B_lora_single_device
code_llama2/7B_qlora_single_device
llama3/8B_lora_single_device
llama3_1/8B_lora_single_device
llama3/8B_qlora_single_device
llama3_1/8B_qlora_single_device
llama2/13B_qlora_single_device
mistral/7B_lora_single_device
mistral/7B_qlora_single_device
gemma/2B_lora_single_device
gemma/2B_qlora_single_device
gemma/7B_lora_single_device
gemma/7B_qlora_single_device
phi3/mini_lora_single_device
phi3/mini_qlora_single_device
qwen2/7B_lora_single_device
lora_dpo_single_device llama2/7B_lora_dpo_single_device
lora_dpo_distributed llama2/7B_lora_dpo
lora_finetune_distributed llama2/7B_lora
llama2/13B_lora
llama2/70B_lora
llama3/70B_lora
llama3_1/70B_lora
llama3/8B_lora
llama3_1/8B_lora
mistral/7B_lora
gemma/2B_lora
gemma/7B_lora
phi3/mini_lora
qwen2/7B_lora
lora_finetune_fsdp2 llama2/7B_lora
llama2/13B_lora
llama2/70B_lora
llama2/7B_qlora
llama2/70B_qlora
generate generation
eleuther_eval eleuther_evaluation
quantize quantization
qat_distributed llama2/7B_qat_full
llama3/8B_qat_full
Personally, I find this almost impossible to parse - my eyes go all blurry trying to scan through this to find a config I might be interested in.
This is scaling quickly with new models and new recipes and I'd like to open up discussion on how we can improve UX here. Will add an idea I have soon.
this is another discussion, but i wonder if we should delete qlora configs, and just make it a flag in lora configs, so we can reduce the amount of configs
I recall @joecummings mentioning we can call tune ls <recipe>
instead of listing all recipes. We can also split into a "models" column and a "config" column so it's slightly easier to read
this is another discussion, but i wonder if we should delete qlora configs, and just make it a flag in lora configs, so we can reduce the amount of configs
This is a really interesting point. My very radical suggestion here was going to be storing only template configs, and forcing users to always specify config parameters which aren't common across models (tokenizer, model, et cetera). This is crazy though.
We're balancing keeping things clean and easily discoverable/accessible, with offering sufficient abstractions to allow the lowest-common-denominator technical user to use our features. Right now, the surface area of no-code pathways for our features is pretty good, so I'd suggest we think about when we're reducing that and asking the user to figure things out by themselves.
I recall @joecummings mentioning we can call tune ls
instead of listing all recipes. We can also split into a "models" column and a "config" column so it's slightly easier to read
I was thinking along similar lines. I'd looove to see some fuzzy matching here, so we might have:
tune ls
by default requires some entity to query over, either a config or model, exempli gratia; tune ls llama2
, tune ls lora
, tune ls llama2_7b
, tune ls single_device
et cetera.tune ls
as is is replaced by tune ls --all
and formatted nicerbumping https://github.com/pytorch/torchtune/pull/1163 as relevant here. subsequently ccing @pbontrager for thoughts
Currently,
tune ls
is a bit unweildy. Can we make it better?@joecummings