Adding Custom Evaluation Job

[fratajcz]

Hi again!

I have written my own evaluation job which worked a few commits back when I was just adding it to kge/job/__init__.py and registering it in kge/job/eval.py. However, now I wanted to add it "properly" by giving it a yaml file and importing it for import or modulesfrom my search job yaml file.

Assume my file is called MyFile.py, which implements class MyJob(EvaluationJob), which I have put into kge/job/. It is accompanied by a file called MyFile.yaml, which contains:

import: [eval]

MyFile:
  class_name: MyJob
  base_job:
    type: EvaluationJob

However, I have no clue how a yaml file should look like for a job, since the jobs that are implemented don't have yaml files.

The Error Message I am getting is:

"""
Traceback (most recent call last):
  File "/home/user/kge/kge/config.py", line 56, in get
    result = result[name]
KeyError: 'MyJob'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/kge/kge/config.py", line 93, in get_default
    parent_type = self.get(parent + "." + "type")
  File "/home/user/kge/kge/config.py", line 58, in get
    raise KeyError(f"Error accessing {name} for key {key}")
KeyError: 'Error accessing MyJob for key MyJob.type'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/anaconda3/envs/kge/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/home/user/kge/kge/job/search.py", line 232, in _run_train_job
    raise e
  File "/home/user/kge/kge/job/search.py", line 140, in _run_train_job
    job = Job.create(
  File "/home/user/kge/kge/job/job.py", line 82, in create
    return TrainingJob.create(
  File "/home/user/kge/kge/job/train.py", line 129, in create
    return init_from(
  File "/home/user/kge/kge/misc.py", line 38, in init_from
    return getattr(module, class_name)(*args, **kwargs)
  File "/home/user/kge/kge/job/train_negative_sampling.py", line 19, in __init__
    super().__init__(
  File "/home/user/kge/kge/job/train.py", line 101, in __init__
    self.valid_job = EvaluationJob.create(
  File "/home/user/kge/kge/job/eval.py", line 40, in create
    class_name = config.get_default(f"{eval_type}.class_name")
  File "/home/user/kge/kge/config.py", line 103, in get_default
    raise e
  File "/home/user/kge/kge/config.py", line 83, in get_default
    return self.get(key)
  File "/home/user/kge/kge/config.py", line 58, in get
    raise KeyError(f"Error accessing {name} for key {key}")
KeyError: 'Error accessing MyJob for key MyJob.class_name'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/anaconda3/envs/kge/bin/kge", line 33, in <module>
    sys.exit(load_entry_point('libkge', 'console_scripts', 'kge')())
  File "/home/user/kge/kge/cli.py", line 285, in main
    job.run()
  File "/home/user/kge/kge/job/job.py", line 159, in run
    result = self._run()
  File "/home/user/kge/kge/job/search_auto.py", line 160, in _run
    self.submit_task(
  File "/home/user/kge/kge/job/search.py", line 75, in submit_task
    self.wait_task()
  File "/home/user/kge/kge/job/search.py", line 97, in wait_task
    self.ready_task_results.append(task.result())
  File "/home/user/anaconda3/envs/kge/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/home/user/anaconda3/envs/kge/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
KeyError: 'Error accessing MyJob for key MyJob.class_name'

I get that i doesnt find the key type, but I definitely did set the key class_name. In fact I am clueless on what the yaml should contain and where it should be placed.

Thanks again for any clues!

EDIT:

I import my class in kge/job/__init__pyby adding from kge.job.MyFile import MyJob and import it in my search job yaml file by adding MyJobto the import parameter

[fratajcz]

Nevermind, I just changed my MyFile.yaml to:

MyJob:
  class_name: MyJob
  type: EvaluationJob

and it works now! Sometimes less is more :) Maybe someone who has the same question will read this.