Closed lorenzomammana closed 3 months ago
Thank you for your contribution, I will review the code recently
Mmm ok, I see that my design is currently disaligned with the current implementation, I can update the PR with your suggestions.
I still really don't like the fact that it's required to specify extra arguments to the CLI every time a user wants to include extra tasks or models, maybe both ways could be included? For the CLI I see as advantage not requiring building a package around tasks and model
From my point of view including models and tasks in command args would be a more explicit way and align with current code style better. I believe would be okay to keep both as current plugin implementation won't affect the pipeline.
@Luodian , Do you mind check this PR also? Which code style do you prefer?
From my point of view including models and tasks in command args would be a more explicit way and align with current code style better. I believe would be okay to keep both as current plugin implementation won't affect the pipeline.
@Luodian , Do you mind check this PR also? Which code style do you prefer?
Thanks for this PR! I take into the look and feel good to merge this.
Nice! I was working exactly right now, I've included a parameter also for the CLI, still believe that's not a good option π
This is an example of the infinite command π
accelerate launch --num_processes=8 -m lmms_eval --model textmonkey --tasks chords --batch_size 1 \
--log_samples --log_samples_suffix cogvlm_chords --output_path ./logs/ \
--include_path=/teamspace/studios/this_studio/lvlm-plugin/lvlm_benchmark/tasks/chords/ \
--include_model=/teamspace/studios/this_studio/lvlm-plugin/lvlm_benchmark/models/textmonkey.py:TextMonkey`
I've arrived just 5 minutes too late haha, good I didn't like the last implementation too much π
Thanks for your work!
Take time, you can still send new PR~
Repost of #102
At the current stage there's a somehow simple way to include new tasks using the --include_external flag, yet there's no way to include external models except from cloning the lmms-eval repository and doing modifications on it, which might not be the ideal scenario in many cases (for example when developing new models internally).
In my scenario I would like to work on an external package (let's say it's an lmms-eval plugin) without the necessity of cloning this repository (using it as a package)
Right now I've done a fork of the repository which allows loading external tasks and models using an environment variable called
LMMS_EVAL_PLUGINS
to include external repositories (like LMMS_EVAL_PLUGINS=package1,package2), particularly in the__main__.py
I'm able to integrate new tasks in this wayAssuming that there's a package installed which follows the same structure as lmms-eval, this piece of code retrieves all the content of {package_name}.tasks and includes them as tasks.
In a similar way I've modified the
__init__.py
of modelsIn a similar way assuming that there's an external package that follows the same structure as lmms-eval, this piece of code will read the AVAILABLE_MODELS dict from the external package and register new models to be used by lmms-eval.
I've tested this with an external package named
lvlm_benchmarks
where I'm trying a few models that are not included in this repo (I could contribute with them as well but they are just ugly implementations π)The
__init__.py
of models contains simply