search

qcri / LLMeBench

Benchmarking Large Language Models
76 stars 15 forks source link

Add a new `assets` subcommand to manage assets. #329

Closed fdalvi closed 1 month ago

fdalvi commented 1 month ago

This commit adds a new assets subcommand to manage various asset related functions. Currently one function is implemented, which is downloading/updating latest assets. This is done by running

python -m llmebench assets download

which downloads assets from our git repo in the current working directory. This command also accepts an optional --work_dir argument, which controls where the assets are downloaded (they will appear in <work_dir>/assets, along with a hidden <work_dir>/.git that tracks the assets).

For consistency sake, I've also moved the older "download" command to be under a dedicated "data" subcommand.

So:

python -m llmebench download ArSAS

now becomes

python -m llmebench data download ArSAS
fdalvi commented 1 month ago

Happy to discuss the naming of the specific commands btw, currently our help looks like:

usage: __main__.py [-h] {benchmark,data,assets} ...

positional arguments:
  {benchmark,data,assets}
                        Defaults to 'benchmark'. Specify a command before the help flag to see detailed usage for each command.
    benchmark           Run the benchmark
    data                Dataset specific commands
    assets              Assets specific commands

optional arguments:
  -h, --help            show this help message and exit

With benchmark being the default if no command is specified.

This change is specifically for our pip release, since the pip package will not contain the assets (even if the do, the user will have to point to some weird directory install python installed packages which is not what we want).