Feedback on Validation Code for LAP Challenge Task 1

[yzyouzhang]

Hi LAP Challenge organizers,

Thank you for organizing this interesting challenge! I am currently participating in Task 1 and have encountered potential issues in the validation code. The issues are related to the script available at: https://github.com/Audio-Experience-Design/LAPChallenge/blob/main/task1/task1_validate.m.

1. Upon reviewing lines #72 and #76, it appears that the condition where metrics_diff equals the thresholds is not addressed, potentially resulting in "0 out of 6 metrics are invalid", which reads confusing.
2. The current script does not detect NaN values from the calculated metrics_diff, requiring participants to manually identify and address these anomalies, which could be streamlined for a smoother validation process.

I believe addressing these issues would enhance the validation process, making it more robust and fair for all participants. Thank you for considering these suggestions, and I look forward to any updates you might implement.

[rapolasd]

Hi You,

Thank you for your comment and for participating in the challenge!

We have updated the validation code for Task 1, to include an assertion on NaN values in the metrics and we have fixed the code behavior when the difference in metrics is equal to thresholds. It should now correctly count those metrics as not meeting the threshold criteria. Anyways, the exact details of metric differences can be be investigated after running the validation code and comparing the values of metrics_diff in the workspace to the thresholds from line #71.

Since metrics_diff is calculated from the outputs of the auditory model, I'd be curious to know in what cases you managed to get NaN values for the metrics?

Let us know if you have any other issues with the code.