How to use customize assessor in NNI?

[skyling0299]

Describe the issue: How to use customized assessor in NNI? I can run my experiment when using builtin accessors like Medianstop. But when I want to customize my own accessors, it starts having problems. i learn from this: https://nni.readthedocs.io/en/stable/hpo/custom_algorithm.html and set the config.yml as the manual. but when i run the command: nnictl create --config my_config_path/config.yml --port my_customize_port i got the following error:

Traceback (most recent call last):
  File "/opt/conda/envs/Fusion_trans/bin/nnictl", line 8, in <module>
    sys.exit(parse_args())
  File "/opt/conda/envs/Fusion_trans/lib/python3.8/site-packages/nni/tools/nnictl/nnictl.py", line 503, in parse_args
    args.func(args)
  File "/opt/conda/envs/Fusion_trans/lib/python3.8/site-packages/nni/tools/nnictl/launcher.py", line 91, in create_experiment
    exp.start(port, debug, RunMode.Detach)
  File "/opt/conda/envs/Fusion_trans/lib/python3.8/site-packages/nni/experiment/experiment.py", line 135, in start
    self._start_impl(port, debug, run_mode, None, [])
  File "/opt/conda/envs/Fusion_trans/lib/python3.8/site-packages/nni/experiment/experiment.py", line 94, in _start_impl
    config = self.config.canonical_copy()
  File "/opt/conda/envs/Fusion_trans/lib/python3.8/site-packages/nni/experiment/config/base.py", line 166, in canonical_copy
    canon._canonicalize([])
  File "/opt/conda/envs/Fusion_trans/lib/python3.8/site-packages/nni/experiment/config/experiment_config.py", line 121, in _canonicalize
    _AlgorithmConfig(**algo)  # pylint: disable=not-a-mapping
  File "/opt/conda/envs/Fusion_trans/lib/python3.8/site-packages/nni/experiment/config/base.py", line 98, in __init__
    raise AttributeError(f'{class_name} does not have field(s) {fields}')
AttributeError: _AlgorithmConfig does not have field(s) codedir, classfilename

i am confused how to fix the problem?

Environment:

NNI version: 2.10 Training service (local|remote|pai|aml|etc): local Client OS: Ubuntu 18.04 Server OS (for remote mode only): Python version: 3.8.16 PyTorch/TensorFlow version: Pytorch 1.10.1 Is conda/virtualenv/venv used?: virtualenv in anaconda Is running in Docker?: no Configuration:

Experiment config (remember to remove secrets!):

searchSpaceFile: search_space.json

trialCommand: python3 ~/my_path/train.py --my_args my_args

trialConcurrency: 2
trialGpuNumber: 4

maxTrialNumber: 100

maxExperimentDuration: 999h

experimentWorkingDirectory: "../result/"

tuner:
  name: TPE
  classArgs:
    optimize_mode: minimize

assessor:
  codeDir: /local/nni/
  classFileName: Assessor_test.py
  className: CustomizedAssessor
  # Any parameter need to pass to your Assessor class __init__ constructor
  # can be specified in this optional classArgs field, for example
  classArgs:
    epoch_num: 40
    start_up: 10
    gap: 5
    higher_is_better : True

trainingService:
  platform: local
  useActiveGpu: true

Log message:

no log message, i can not even run the experiment.

Thanks for any suggestions

### Tasks

[MischaPanch]

The documentation is currently wrong, I wanted to open a PR to fix it. The correct format is:

  codeDirectory: ...
  className: <module>.<class_name>
  classArgs:
      ....

However, in my experiments the custom assessor is not properly working and there is no way to debug, of course.

I started to have a general question - is this project being abandoned? There is not much activity in commits, all kinds of things in the docs are wrong, and many links are broken. The pre-release happened a while ago, but there seems to be no timeline on the actual release. I found several things to be quite buggy of flaky... It would be a shame, I really like the promise of nni.