Single entry point script

[malteos]

Merge the different task scripts into a single entry point script.

Subtask:

• [x] Single main.py script to run evaluation on different task types
• [x] Multiple tasks can be evaluated at once. Provide a --task_list <task_a>,<task_b>,... argument.
• [x] Each task is only identified with its name without the need for specifying task type, label column etc. Store these information in a separate python file/class as done in https://github.com/OpenGPTX/lm-evaluation-harness/blob/master/lm_eval/tasks/__init__.py

[malteos]

Example:

python main.py --model MY_NEW_MODEL --task_list ALL

[malteos]

Some feedback:

• The main.py is super hacky. Calling other main functions within a main is bad practice.
• This seems also hacky https://github.com/malteos/finetune-evaluation-harness/blob/main/tasks/germeval2017.py#L39-L78
• Each sub-main has a lot of duplicated code. This makes changing stuff complicated (as you probably saw when adding peft)
• Try to remove all duplicated code. There are tools to check this. https://marketplace.visualstudio.com/items?itemName=jianbingfang.dupchecker
• PEFT should have its own argument class like https://github.com/malteos/finetune-evaluation-harness/blob/main/hf_scripts/model_args.py
• Do not set important hyperparameters somewhere in the middle of the code as in https://github.com/malteos/finetune-evaluation-harness/blob/main/hf_scripts/hgf_fine_tune_class.py#L312-L313 (use arguments instead)
• Use class abstraction for tasks https://github.com/malteos/finetune-evaluation-harness/blob/main/tasks/germeval2017.py#L17

TODOs

• Refactor the code base
• Integrate unit tests. For example with pytest https://docs.pytest.org/en/7.2.x/
• Measure test coverage with https://coverage.readthedocs.io/en/latest/ (coverage should be at least +80% better +90%)
• Check code formatting/style https://github.com/psf/black https://github.com/PyCQA/pylint
• Update the README

[malteos]

Feedback II

• README:
  • Please include more details on the available arguments (maybe just the --help output?)
  • Please link the datasets to their homepages
• main/process_args: No need for two files since main.py is only calling process args
  • is this necessary? https://github.com/malteos/finetune-evaluation-harness/blob/main/process_args.py#L19-L24 cannot you just include InitialArguments in line 16?
  • please use the logging classes and not print statements. See https://realpython.com/python-logging/

• tests:

  • for unit tests, you do not need to load actual large models. better create a dummy model that is very tiny so testing does not take too long
  • the assertion test on the return type = Trainer only tests whether the code runs all the way to the end but does not check if the actual output is correct.
  • better test on the actual output, e.g., eval score = xy
  • tests regarding loading the models/configs are unnecessary as they basically replicated HF transformers -> only test for logic that you implemented
  • task classes:

  • the getter functions can be part of the parent class since they are the same for all tasks:

    
    class Classification(object):
    DATASET_ID = None
    TASK_NAME = None
    LABEL_NAME = None
    def get_dataset_id(self):
       return self.DATASET_ID
    
    #  ....

class GermEval2018(Classification): DATASET_ID = "philschmid/germeval18" # HF datasets ID TASK_NAME = "germeval2018" LABEL_NAME = "multi" # column name from HF dataset