search

IntelLabs / Hardware-Aware-Automated-Machine-Learning

MIT License
14 stars 4 forks source link

Searching Code is missing for commonsense and math tasks #11

Open pprp opened 2 weeks ago

pprp commented 2 weeks ago

Thank you for your great work and swift answers. Maybe LoNAS (or LLaMA-NAS) can bring one-shot NAS back to the era of LLMs.

I noticed that in run_glue.py, when we set the flag do_search=True, we have the following code:

https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/blob/a00c14855ed87a124602ea89686dc94e8f3372f4/LoNAS/run_glue.py#L701-L752

However, I can not find similar code in commonsense or math:

https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/blob/a00c14855ed87a124602ea89686dc94e8f3372f4/LoNAS/run_commonsense.py#L327-L337

Can you explain it?

Yuan0320 commented 2 weeks ago

Thanks for your invaluable feedback! We have added the code for LoNAS to search on the math and commonsense datasets: https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning/commit/f6cd4554b256ddf7ac431c10a2c59e4f635198bf.

Running an evolutionary search on LLMs is expensive, so you may want to propose other ways to identify high-performing subnetworks. We are also exploring this challenge.

We invite you to check our latest work, Shears (paper, code), in which we constrain the search space to the hyper-parameters of elastic LoRA adapters. You might find this work useful to your research.

Thanks!