evaluation

[arunodhayan]

when trying python3 task1b.py -m eval -o output.csv

The evaluation script saves an empty CSV file and doesn't print any evaluation results

[toni-heittola]

In the evaluation mode, no evaluation results are calculated as the evaluation dataset does not come with reference labels. The evaluation results will be provided through DCASE Challenge. On the other hand, the CSV file should not be empty, as it should contain system estimates ready for Challenge submission. The CSV file saving is done in "Testing stage", and the system should output as the last line:

[I] System output saved to [output.csv]

Check that you have the evaluation dataset downloaded, and extracted in the path dcase2020_task1_baseline/datasets/TAU-urban-acoustic-scenes-2020-3class-evaluation. For example, you should have 8641 files in dcase2020_task1_baseline/datasets/TAU-urban-acoustic-scenes-2020-3class-evaluation/audio.

Is your system working normally in the dev mode?

python3 task1b.py -m dev or the same thing shorter python3 task1b.py

Or with python3 task1b.py -m dev -o output.csv

[arunodhayan]

it works well for dev i could visualize the output but not with eval

[toni-heittola]

Hard to debug with such a small amount of information. I just ran the code in my setup, and it working as intended.

Did you manage to train a model in the eval mode, using all data from the development dataset?

[arunodhayan]

yes I did I tried in eval mode and I will try once again and update you.

[toni-heittola]

By looking at code, one can get an empty CSV file only when the dataset class does not return any test files. I added now double-check for this with commit d62573467695f0d6a133651480529db804544374 and now system will raise a ValueError in this case.

You should have a file dcase2020_task1_baseline/datasets/TAU-urban-acoustic-scenes-2020-3class-evaluation/evaluation_setup/fold1_test.csv, in size of 134K, which is used in eval mode to go through all evalaution dataset files.

[arunodhayan]

i had checked the file size it has 134kb -rw-r--r-- 1 root root 137140 Jun 11 18:40 fold1_test.csv

[arunodhayan]

[I] [ 8633 / 8640] [993.wav] [I] [ 8634 / 8640] [994.wav] [I] [ 8635 / 8640] [995.wav] [I] [ 8636 / 8640] [996.wav] [I] [ 8637 / 8640] [997.wav] [I] [ 8638 / 8640] [998.wav] [I] [ 8639 / 8640] [999.wav] [I] DONE [0:09:12.235721] [8640 items] [0:00:00.063916 per item] [I] [I] Testing [I] ======================================== [I] Fold [all_data] [I] DONE [0:00:02.220386] [I] [I] System output saved to [output.csv] [I] root@TitanX:/src# cat output.csv filename scene_label empty file

[toni-heittola]

How are other stages in the eval mode, the following modes should be there:

• Feature Extraction, this is for all data in the development dataset ('all_data' fold mode)
• Feature Normalization, this is for all data in the development dataset ('all_data' fold mode)
• Learning, this is using all data from the development dataset ('all_data' fold mode)
• Feature Extraction, this is for all data in the evaluation dataset
• Testing, this is for all data in the evaluation dataset

[arunodhayan]

yes it is happening as per order am getting the output as empty CSV file

[toni-heittola]

OK, then it seems that we are out of luck with remote debugging. I think best is that you insert

from IPython import embed
embed()

inside do_testing (maybe at the end https://github.com/toni-heittola/dcase2020_task1_baseline/blob/master/task1a.py#L1069) function and see why output file is empty. Even though we are debugging task1b system, the do_testing function is shared across subtasks, and because of this, you need to edit task1a.py file. Thinks to check:

1. Is model loaded correctly https://github.com/toni-heittola/dcase2020_task1_baseline/blob/master/task1a.py#L985
2. Are features loaded correctly https://github.com/toni-heittola/dcase2020_task1_baseline/blob/master/task1a.py#L1008
3. Is it shaped correctly https://github.com/toni-heittola/dcase2020_task1_baseline/blob/master/task1a.py#L1010
4. Do you get probabilities https://github.com/toni-heittola/dcase2020_task1_baseline/blob/master/task1a.py#L1025
5. Or result per file https://github.com/toni-heittola/dcase2020_task1_baseline/blob/master/task1a.py#L1052
6. Do you have stuff in res MetaDataContainer, with e.g. res.log()

[arunodhayan]

i cloned the repository as new will trying running the base script once and try this method

[arunodhayan]

i had cloned the repository newly and tried to run it in eval mode still my csv file is empty

[toni-heittola]

Did you go through the check-list I gave in the last post? Where does it fail?

[arunodhayan]

till probabilities its working file

i tried res.log and tried to print res i displays empty lines

[toni-heittola]

So are you getting probabilities out? And what happens between lines 1025 and 1052?

[arunodhayan]

yes, probabilities are out but the file is not getting saved only for eval mode in case of dev mode I could save the CSV file and could visualize it .........

[toni-heittola]

The do_testing function will load results from the disk if they have been already calculated. I could be that while testing you have managed to create empty results file, and system is reusing it.

You can retrigger results file generation by deleting files under dcase2020_task1_baseline/system/task1b/recognizer (this will also delete results from dev mode). You can also use overwrite flag to retrigger the results calculation (this will also retrain the model, to avoid that modify the task1b.yaml and flow section there to skip learning).

The actual CSV file saving happens in save_system_output function.

I added now a check for empty results file in commit 22cd37c1b602e711c86b1f8d922cc52b753d3ae7

So you could next check that results are read in fully here > https://github.com/toni-heittola/dcase2020_task1_baseline/blob/master/utils.py#L259

[arunodhayan]

is these path is an hidden file path system/task1b/recognizer

[toni-heittola]

Correction, in eval model directory eval_recognizer is used (system/task1b/eval_recognizer), just rename that to something else for the debuggin purposes.

[arunodhayan]

mode='dcase' File "/src/utils.py", line 247, in save_system_output output_file=output_file ValueError: There are no results to output into [out.csv]

[arunodhayan]

in eval mode, do_testing function is not happening, I wantedly made an error res_data in do_testing in eval mode, it doesn't throw me any error as expected in dev mode it threw me an error

[toni-heittola]

I found some inconsistency how evaluation dataset were constructed in dcase_util. I fixed these and made a new version, so update dcase_util to version 0.2.15 and test again.

[arunodhayan]

i found out that in do_testing in eval mode input data is not being read line 1008 input_data = features.data i tried to print no data is being printed

[toni-heittola]

But you already verified that propabilities are valid and those features are need for that?

[arunodhayan]

I made an error just checked log files I checked it in dev mode sorry for that

now switched to eval mode and checked again where i found out input data for eval mode is not being read

[arunodhayan]

it is working now after the dcase_util==0.2.15

[toni-heittola]

Nice!