ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.

[haoshuai714]

When I directly used the pre trained model test, an error occurred:ValueError: Only one class present in y_true. ROC AUC score is not defined in that case. Can I directly use the pre training model provided by github for direct testing?

[JacobChalk]

It seems likely this error is caused from running test_net.py on the test timestamps file, where the dataset gives all samples a fake label of 0, which breaks the evaluation code due to the error stated in this issue. We have just pushed a change to fix this. It's worth noting that metrics on the test set can only be obtained by entering and submitting to the audio-based interaction recognition challenge.

[haoshuai714]

yes， you are right. I ran the command: python tools/run_net.py \ --cfg configs/EPIC-Sounds/slowfast/SLOWFASTAUDIO_8x8_R50.yaml \ TRAIN.ENABLE False \ TEST.ENABLE True \ NUM_GPUS num_gpus \ OUTPUT_DIR /path/to/outpur_dir \ EPICSOUNDS.AUDIO_DATA_FILE /path/to/EPIC_audio.hdf5 \ EPICSOUNDS.ANNOTATIONS_DIR /path/to/annotations \ TEST.CHECKPOINT_FILE_PATH /path/to/experiment_dir/checkpoints/checkpoint_best.pyth \ EPICSOUNDS.TEST_LIST EPIC_Sounds_recognition_test_timestamps.pkl

Can I now use the pre-trained model and code you provided to run the files submitted for the competition?

[JacobChalk]

Yes you should be able to now. The TestMeter has been updated so that it sees if you are running inference on the test split, if you are (and assuming the file names are as they are in this repo), it will not compute stats and only extract predictions when using the test timestamp

[haoshuai714]

In other words, the pre-trained model you provided can be used directly for testing, or for fine-tune in the training phase, right?

[JacobChalk]

Yes, it can be used for both with the current code

[haoshuai714]

After running the test code, I generated the epic_sounds_recognition_test_timestamps.pkl document. However, the official website submission is a JSON file: $ zip -j my-submission.zip test.json. How should I generate related test.json?

[JacobChalk]

The output predictions in the .pkl format can be made into the JSON submission by following the instructions on the challenge repo here.