search

zerospeech / benchmarks

A command line tool that helps use the "Zero Ressource Challenge" benchmarks
https://zerospeech.com/toolbox/
GNU General Public License v3.0
8 stars 2 forks source link

Fix abxLS benchmark crash #8

Closed hallapmark closed 1 year ago

hallapmark commented 1 year ago

The abxLS benchmark crashes.

To reproduce:

conda create -n py39test python=3.9
conda activate py39test
pip install zerospeech-libriabx2
pip install zerospeech-benchmarks[all]
zrc datasets:pull abxLS-dataset
zrc benchmarks:run abxLS [path/to/submission]

This produced the following error:

Traceback (most recent call last):
  File "/home/mhallap/.conda/envs/py39test/bin/zrc", line 8, in <module>
    sys.exit(main())
  File "/home/mhallap/.conda/envs/py39test/lib/python3.9/site-packages/zerospeech/startup.py", line 39, in main
    cli.run()
  File "/home/mhallap/.conda/envs/py39test/lib/python3.9/site-packages/zerospeech/cmd/cli_lib.py", line 258, in run
    cmd.run_cmd(argv=sys.argv[2:])
  File "/home/mhallap/.conda/envs/py39test/lib/python3.9/site-packages/zerospeech/cmd/cli_lib.py", line 91, in run_cmd
    self.run(args)
  File "/home/mhallap/.conda/envs/py39test/lib/python3.9/site-packages/zerospeech/cmd/benchmarks.py", line 74, in run
    if not submission.valid:
  File "/home/mhallap/.conda/envs/py39test/lib/python3.9/site-packages/zerospeech/model/benchmark.py", line 186, in valid
    self.__validate_submission__()
  File "/home/mhallap/.conda/envs/py39test/lib/python3.9/site-packages/zerospeech/benchmarks/_benchmarks/abx_LS/data_model.py", line 68, in __validate_submission__
    self.validation_output = AbxLSSubmissionValidator().validate(self)
  File "/home/mhallap/.conda/envs/py39test/lib/python3.9/site-packages/zerospeech/model/benchmark.py", line 142, in validate
    res = validator(item)
  File "/home/mhallap/.conda/envs/py39test/lib/python3.9/site-packages/zerospeech/model/benchmark.py", line 24, in _impl
    return method(self, *method_args, **method_kwargs)
  File "/home/mhallap/.conda/envs/py39test/lib/python3.9/site-packages/zerospeech/benchmarks/_benchmarks/abx_LS/validators.py", line 38, in validate_dev_clean
    results = validators.numpy_array_list_check(
  File "/home/mhallap/.conda/envs/py39test/lib/python3.9/site-packages/zerospeech/validators/validators.py", line 62, in numpy_array_list_check
    results.extend(fn(item))
TypeError: 'NoneType' object is not iterable
nhamilakis commented 1 year ago

A temporary fix could be to not do the validation of the submission (option: --skip-validation).

Although i will look into the issue on my side and see why it is failing.