Allow loading models and tasks from external packages

At the current stage there's a somehow simple way to include new tasks using the --include_external flag, yet there's no way to include external models except from cloning the lmms-eval repository and doing modifications on it, which might not be the ideal scenario in many cases (for example when developing new models internally).

In my scenario I would like to work on an external package (let's say it's an lmms-eval plugin) without the necessity of cloning this repository (using it as a package)

Right now I've done a fork of the repository which allows loading external tasks and models using an environment variable called LMMS_EVAL_PLUGINS to include external repositories (like LMMS_EVAL_PLUGINS=package1,package2), particularly in the __main__.py I'm able to integrate new tasks in this way

if os.environ.get("LMMS_EVAL_PLUGINS", None):
    for plugin in os.environ["LMMS_EVAL_PLUGINS"].split(","):
        package_tasks_location = importlib.util.find_spec(f"{plugin}.tasks").submodule_search_locations[0]
        eval_logger.info(f"Including path: {args.include_path}")
        include_path(package_tasks_location)

Assuming that there's a package installed which follows the same structure as lmms-eval, this piece of code retrieves all the content of {package_name}.tasks and includes them as tasks.

In a similar way I've modified the __init__.py of models

if os.environ.get("LMMS_EVAL_PLUGINS", None):
    # Allow specifying other packages to import models from
    for plugin in os.environ["LMMS_EVAL_PLUGINS"].split(","):
        m = importlib.import_module(f"{plugin}.models")
        for model_name, model_class in getattr(m, "AVAILABLE_MODELS").items():
            try:
                exec(f"from {plugin}.models.{model_name} import {model_class}")
            except ImportError:
                pass

In a similar way assuming that there's an external package that follows the same structure as lmms-eval, this piece of code will read the AVAILABLE_MODELS dict from the external package and register new models to be used by lmms-eval.

Does it seem something that you can consider to integrate? I could partially contribute with a PR

EvolvingLMMs-Lab / lmms-eval

Allow loading models and tasks from external packages #102