search

QutEcoacoustics / audio-analysis

The audio analysis code (AnalysisPrograms.exe) for the QUT Ecoacoustics Research Group
https://ap.qut.ecoacoustics.info/
Apache License 2.0
52 stars 12 forks source link

Tweaking the recognizers -Testing(JCU) #515

Closed jagakala closed 3 years ago

jagakala commented 3 years ago

Tiltle: Confusion matrix output doesn't reflect any variation despite the change of values in the recognizers

Issue : The training and test egret yml recognizer's output remains the same after modifying the relevant species' yml file.

            For example, barking owl's training egret yml  which is configured with the recognizer 
           "Truskinger.NinoxConnivens.yml". Even after changing the "MinHeartz from 200 to100" and "MaxHeartz from 1150 to 
           500" values, the output doesn't show any variation in the confusion matrix  results with respect to accuracy 
           or any other fields.

Proposal:

Modified parameter values in the respective configuration recognizer gets reflected only if the complete path of the recognizer is given in the test suites configuration of the corresponding species."

Repalce with what you want us to change/add/remove?»

Add the complete file location path of the "Barking Owl" which would read like below "test_suites: 2 Barking owl: tool_configs:

ap: config: C:\AP\ConfigFiles\RecognizerConfigFiles\Truskinger.NinoxConnivens.yml versus the original configuration of** config : Truskinger.NinoxConnivens.yml

Alternatives:

«Tell us: is there any way around the problem currently? Could we choose not to do this to make things simpler? What drawbacks does your suggestion have?»

  1. To improve the efficiency of recognizers, tweaking them is important, and without this modification the AP program doesn't reflect any variation to validate the testing.

  2. As of now, we don't find any other workaround without this change to do testing. However, we think that the AP program's relevant recognizers should be tested to see why the recognizers are still pointed or running with the values of the old recognizer instead of the saved modified recognizer.

atruskie commented 3 years ago

Hi @jagakala,

Confusion matrix output doesn't reflect any variation despite the change of values in the recognizers

AP does not produce a confusion matrix. You might be thinking of https://github.com/QutEcoacoustics/egret

To distill your request: when using the "helper" function of resolving a config file by name (which defaults to a config shipped with AP) it can be confusing which config file is used.

I'm fairly confident that changes in the default config files and any user edited config file are reflected in the program. If you think this is not the case, send me some logs from AP running so I can diagnose.

I do need to reconsider how we work with config files - I'll keep this feedback in mind for that.

Since I don't think there is a bug here, I'm going to close this issue. If you think there is still a bug, then send logs from an example run of AP where it happens, and reopen.

Perhaps we just need to update our docs on config files? https://ap.qut.ecoacoustics.info/basics/config_files.html