search

openai / mle-bench

MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineering
https://openai.com/index/mle-bench/
Other
534 stars 60 forks source link

Improve Efficiency and Error Handling in `mlebench/cli.py` #12

Closed Mefisto04 closed 1 month ago

Mefisto04 commented 1 month ago

The current implementation in mlebench/cli.py has areas that can be improved for better efficiency and error handling:

  1. Error Handling:

    • Add checks for invalid competition IDs or missing arguments, providing informative error messages.
    • Wrap file operations (e.g., reading a list of competitions from a file) in try-except blocks to handle potential IOError or FileNotFoundError.
    • Ensure that new_registry.get_competition() returns a valid competition object to avoid potential AttributeError.
    • Include a check for unsupported commands to handle cases where args.command does not match any expected values.

  2. Efficiency:

    • Avoid repeated calls to registry.list_competition_ids() by storing the result in a variable when used multiple times.
    • Parallelize downloads and other operations in the prepare and download-leaderboard commands to speed up processing, especially when working with multiple competitions.

Implementing these improvements will enhance the performance of the code. Please assign me this issue so that i can contribute in it.

Mefisto04 commented 1 month ago

hey @thesofakillers , please check this once.

thesofakillers commented 1 month ago

Hi, the CLI works well enough and is certainly not the bottleneck for MLE-bench, so I am closing this issue.

As for parallelized downloads, I believe this risks running into rate limits and overwhelming kaggle, so i think its fine to leave as is.