Classification Benchmark and false positives

ghost commented 3 years ago

Hello, I was able to reproduce and run the Lite System on a MacBook M1 and also on a Raspberry Pi with the live system from Patrik. I run a lot of benchmark tests with Germany Species which are transcribed (at least with there German names) available on YouTube. I used also the German localization provided in with the new labels file. The classification result (output labels) was on both systems more less the same. I used the following Input sound files:

I transformed the YouTube videos to mp3 and used my home location (Rhineland Pfalz) as parameter for analyse script. I did not support and week of the year flag. The results i get where really bad, i get approximately:

True Positives = ~2%
False Positives = 98%

One interesting fact that even trough I specified the geo coordinates (lat/long) from Germany most species are from South-America and the South-East Pacific area. I want to discuss the problems with the classification here in detail:

Did I have some technical problems ?
I compared the German labels file, but the Latin names seems to be the same as in the one supported with the model.
Birds did imitate each other?
There is something wrong with the coordinate settings?
The missing week of the year has much influence on the recognition.
The input Coropora was bad (what i not expect)
Other things what I don't know?

UPDATE --> I tried the iPhone app is working 100%, so i expect a technical Problem here !!!

DD4WH commented 3 years ago

There is not enough information to be able to know what is going on. Please post

your coordinates [if you are unsure about your coordinates, go here: https://www.latlong.net/]
the classification results (species names and confidence scores) It is unclear to me what you did exactly: How can you use BirdNET-Lite without specifying week and why would you do that???

ghost commented 3 years ago

UPDATE --> I tried the iPhone app is working 100%, so i expect a technical Problem here !!!

Here are my coordinates:

LATITUDE=49.584 LONGITUDE=6.888

I didn't give the command line parameter for the week, the one liner runs nevertheless. Only for testing purposes, I don't have a microphone.

ghost commented 3 years ago

Thats what i get for the "Amsel":

Amsel.mp3.zip

Start (s);End (s);Scientific name;Common name;Confidence
0.0;3.0;Heliomaster longirostris;Rosenkehlkolibri;0.937234
3.0;6.0;Thraupis cyanocephala;Blue-capped Tanager;0.38057935
3.0;6.0;Thryophilus pleurostictus;Akazienzaunkönig;0.11324725
6.0;9.0;Nesoptilotis leucotis;Schwarzkehl-Honigfresser;0.19928794
6.0;9.0;Columbina inca;Inkatäubchen;0.13027596
12.0;15.0;Thraupis cyanocephala;Blue-capped Tanager;0.22888827
18.0;21.0;Thraupis cyanocephala;Blue-capped Tanager;0.47750905
18.0;21.0;Melaenornis edolioides;Senegaldrongoschnäpper;0.30733746
21.0;24.0;Theristicus caudatus;Weißhalsibis;0.11770274
24.0;27.0;Himatione sanguinea;Apapanekleidervogel;0.38964814
24.0;27.0;Myrmecocichla monticola;Bergschmätzer;0.21705064
27.0;30.0;Thraupis cyanocephala;Blue-capped Tanager;0.98378444
27.0;30.0;Thryophilus rufalbus;Rotrückenzaunkönig;0.27190888
27.0;30.0;Thamnophilus nigriceps;Schwarzameisenwürger;0.20240785
27.0;30.0;Thamnophilus torquatus;Rostflügel-Ameisenwürger;0.13817915
30.0;33.0;Centropus celebensis;Celebeskuckuck;0.3895321
33.0;36.0;Thraupis cyanocephala;Blue-capped Tanager;0.4682627
33.0;36.0;Melaenornis edolioides;Senegaldrongoschnäpper;0.25869012
33.0;36.0;Thinocorus rumicivorus;Zwerghöhenläufer;0.11964027
36.0;39.0;Thraupis cyanocephala;Blue-capped Tanager;0.19553971
39.0;42.0;Thraupis cyanocephala;Blue-capped Tanager;0.66366124
39.0;42.0;Thamnophilus unicolor;Einfarbameisenwürger;0.21413608
39.0;42.0;Thryophilus pleurostictus;Akazienzaunkönig;0.19057931
39.0;42.0;Thinocorus rumicivorus;Zwerghöhenläufer;0.122284256
45.0;48.0;Tringa flavipes;Gelbschenkel;0.21151997
45.0;48.0;Saltator maximus;Buntkehlsaltator;0.14283802
48.0;51.0;Thraupis cyanocephala;Blue-capped Tanager;0.403783

The iPhone BirdNet.app recognizes the correct bird "Amsel".

DD4WH commented 3 years ago

The easiest explanation would be that you swapped LAT and LON . . .

ghost commented 3 years ago

Nope:

python analyze.py --i ~/Music/SoundAndWaveFileExamples/Bird\ Sounds/Amsel.mp3 --lat 49.584 --lon 6.888 -week 11; cat result.csv

I exchanged LAT LOG and get the same result ?!?

ghost commented 3 years ago

Can you reproduce with the attached file with BirdNET-Lite and BirdNET.app and compare?

DD4WH commented 3 years ago

Please tell us exactly what you installed. Many of the species names you get in your results are not in the labels.txt file. So I can tell from your output that you are not using BirdNET-Lite . . .

ghost commented 3 years ago

I installed the repro fom here, and exchanged the labels.txt from the model to the german one.

DD4WH commented 3 years ago

I would propose you make the base install and test that. After that is working, you can test with other files. And I would propose you use exactly the examples that Stefan gave in his repo for testing. BTW you do not specify an output file and you do not specify week etc. Please use the base install and the example.

ghost commented 3 years ago

Did you get the same Latin names for the examples from Stefan?

(base) ➜  BirdNET-Lite git:(main) ✗ python analyze.py --i 'example/XC558716 - Soundscape.mp3' --lat 35.4244 --lon -120.7463; cat result.csv 
Init Plugin
Init Graph Optimizer
Init Kernel
INFO: Created TensorFlow Lite delegate for select TF ops.
Metal device set to: Apple M1
INFO: TfLiteFlexDelegate delegate: 1 nodes delegated out of 182 nodes with 1 partitions.

LOADING TF LITE MODEL... DONE!
READING AUDIO DATA... /opt/miniforge3/lib/python3.9/site-packages/librosa/core/audio.py:165: UserWarning: PySoundFile failed. Trying audioread instead.
  warnings.warn("PySoundFile failed. Trying audioread instead.")
DONE! READ 46 CHUNKS.
ANALYZING AUDIO... DONE! Time 1.1 SECONDS
WRITING RESULTS TO result.csv ... DONE! WROTE 25 RESULTS.
Start (s);End (s);Scientific name;Common name;Confidence
6.0;9.0;Cranioleuca pyrrhophia;Streifenscheitel-Baumschlüpfer;0.19843541
6.0;9.0;Antrostomus rufus;Rostnachtschwalbe;0.15303311
6.0;9.0;Antrostomus vociferus;Schwarzkehl-Nachtschwalbe;0.105004825
9.0;12.0;Furnarius minor;Kleintöpfer;0.2526896
12.0;15.0;Telophorus sulfureopectus;Sulphur-breasted Bushshrike;0.27489114
18.0;21.0;Furnarius minor;Kleintöpfer;0.113345794
24.0;27.0;Telophorus sulfureopectus;Sulphur-breasted Bushshrike;0.20446834
24.0;27.0;Cinclus pallasii;Pallaswasseramsel;0.14035383
39.0;42.0;Dicrurus adsimilis;Trauerdrongo;0.49781206
42.0;45.0;Dicrurus adsimilis;Trauerdrongo;0.16531901
48.0;51.0;Phoenicurus moussieri;Diademrotschwanz;0.56472397
60.0;63.0;Dicrurus adsimilis;Trauerdrongo;0.32988507
60.0;63.0;Phyllomyias burmeisteri;Burmeisterkleintyrann;0.1034713
63.0;66.0;Dicrurus adsimilis;Trauerdrongo;0.12177477
69.0;72.0;Dicrurus adsimilis;Trauerdrongo;0.24120894
69.0;72.0;Phyllomyias burmeisteri;Burmeisterkleintyrann;0.14926776
84.0;87.0;Riparia chinensis;Graukehl-Uferschwalbe;0.13066193
87.0;90.0;Tringa incana;Wanderwasserläufer;0.29393217
93.0;96.0;Riparia chinensis;Graukehl-Uferschwalbe;0.20919465
96.0;99.0;Eclectus roratus;Edelpapagei;0.35672444
96.0;99.0;Dicrurus adsimilis;Trauerdrongo;0.15679923
114.0;117.0;Furnarius minor;Kleintöpfer;0.13333207
123.0;126.0;Synallaxis brachyura;Graurücken-Dickichtschlüpfer;0.12138738
129.0;132.0;Riparia chinensis;Graukehl-Uferschwalbe;0.12164671
132.0;135.0;Troglodytes troglodytes;Zaunkönig;0.13889167

ghost commented 3 years ago

I cannot compare the examples if i don't have a reference, anyone here who can post the results for the shipped XC... files?

DD4WH commented 3 years ago

Your results are wrong. Use the base install without the WRONG labels.txt file with the German names. If you are lucky, it is only the labels.txt file with the German names that is wrong. If not, make a new install and try out the example. Here you can find details on the example soundscape: https://www.xeno-canto.org/explore?query=nr%3A558716

ghost commented 3 years ago

Plase, can you provide the reference result? The Latin names are the same in the labels file provided here: https://github.com/kahst/BirdNET-Lite/issues/4

ghost commented 3 years ago

Ok, found the Bug the labels file is totally wrong:

(base) ➜  BirdNET-Lite git:(main) ✗ python analyze.py --i 'example/XC558716 - Soundscape.mp3' --lat 35.4244 --lon -120.7463; cat result.csv
Init Plugin
Init Graph Optimizer
Init Kernel
INFO: Created TensorFlow Lite delegate for select TF ops.
Metal device set to: Apple M1
INFO: TfLiteFlexDelegate delegate: 1 nodes delegated out of 182 nodes with 1 partitions.

LOADING TF LITE MODEL... DONE!
READING AUDIO DATA... /opt/miniforge3/lib/python3.9/site-packages/librosa/core/audio.py:165: UserWarning: PySoundFile failed. Trying audioread instead.
  warnings.warn("PySoundFile failed. Trying audioread instead.")
DONE! READ 46 CHUNKS.
ANALYZING AUDIO... DONE! Time 1.1 SECONDS
WRITING RESULTS TO result.csv ... DONE! WROTE 25 RESULTS.
Start (s);End (s);Scientific name;Common name;Confidence
6.0;9.0;Cyanocitta stelleri;Steller's Jay;0.19843541
6.0;9.0;Aphelocoma insularis;Island Scrub-Jay;0.15303311
6.0;9.0;Aphelocoma wollweberi;Mexican Jay;0.105004825
9.0;12.0;Geothlypis tolmiei;MacGillivray's Warbler;0.2526896
12.0;15.0;Troglodytes aedon;House Wren;0.27489114
18.0;21.0;Geothlypis tolmiei;MacGillivray's Warbler;0.113345794
24.0;27.0;Troglodytes aedon;House Wren;0.20446834
24.0;27.0;Cistothorus palustris;Marsh Wren;0.14035383
39.0;42.0;Dryobates nuttallii;Nuttall's Woodpecker;0.49781206
42.0;45.0;Dryobates nuttallii;Nuttall's Woodpecker;0.16531901
48.0;51.0;Pipilo maculatus;Spotted Towhee;0.56472397
60.0;63.0;Dryobates nuttallii;Nuttall's Woodpecker;0.32988507
60.0;63.0;Piranga ludoviciana;Western Tanager;0.1034713
63.0;66.0;Dryobates nuttallii;Nuttall's Woodpecker;0.12177477
69.0;72.0;Dryobates nuttallii;Nuttall's Woodpecker;0.24120894
69.0;72.0;Piranga ludoviciana;Western Tanager;0.14926776
84.0;87.0;Setophaga nigrescens;Black-throated Gray Warbler;0.13066193
87.0;90.0;Vireo cassinii;Cassin's Vireo;0.29393217
93.0;96.0;Setophaga nigrescens;Black-throated Gray Warbler;0.20919465
96.0;99.0;Empidonax occidentalis;Cordilleran Flycatcher;0.35672444
96.0;99.0;Dryobates nuttallii;Nuttall's Woodpecker;0.15679923
114.0;117.0;Geothlypis tolmiei;MacGillivray's Warbler;0.13333207
123.0;126.0;Thryomanes bewickii;Bewick's Wren;0.12138738
129.0;132.0;Setophaga nigrescens;Black-throated Gray Warbler;0.12164671
132.0;135.0;Xanthocephalus xanthocephalus;Yellow-headed Blackbird;0.13889167
(base) ➜  BirdNET-Lite git:(main) ✗ cp ../BirdNET-Lite/model/labels.txt model/labels.txt                                                   
cp: model/labels.txt and ../BirdNET-Lite/model/labels.txt are identical (not copied).
(base) ➜  BirdNET-Lite git:(main) ✗ cp -v ../BirdNET-Lite/model/labels.txt model/labels.txt
cp: model/labels.txt and ../BirdNET-Lite/model/labels.txt are identical (not copied).
(base) ➜  BirdNET-Lite git:(main) ✗ cp -v ../../BirdNET-Lite/model/labels.txt model/labels.txt 
../../BirdNET-Lite/model/labels.txt -> model/labels.txt
(base) ➜  BirdNET-Lite git:(main) ✗ python analyze.py --i 'example/XC558716 - Soundscape.mp3' --lat 35.4244 --lon -120.7463; cat result.csv
Init Plugin
Init Graph Optimizer
Init Kernel
INFO: Created TensorFlow Lite delegate for select TF ops.
Metal device set to: Apple M1
INFO: TfLiteFlexDelegate delegate: 1 nodes delegated out of 182 nodes with 1 partitions.

LOADING TF LITE MODEL... DONE!
READING AUDIO DATA... /opt/miniforge3/lib/python3.9/site-packages/librosa/core/audio.py:165: UserWarning: PySoundFile failed. Trying audioread instead.
  warnings.warn("PySoundFile failed. Trying audioread instead.")
DONE! READ 46 CHUNKS.
ANALYZING AUDIO... DONE! Time 1.1 SECONDS
WRITING RESULTS TO result.csv ... DONE! WROTE 25 RESULTS.
Start (s);End (s);Scientific name;Common name;Confidence
6.0;9.0;Cranioleuca pyrrhophia;Streifenscheitel-Baumschlüpfer;0.19843541
6.0;9.0;Antrostomus rufus;Rostnachtschwalbe;0.15303311
6.0;9.0;Antrostomus vociferus;Schwarzkehl-Nachtschwalbe;0.105004825
9.0;12.0;Furnarius minor;Kleintöpfer;0.2526896
12.0;15.0;Telophorus sulfureopectus;Sulphur-breasted Bushshrike;0.27489114
18.0;21.0;Furnarius minor;Kleintöpfer;0.113345794
24.0;27.0;Telophorus sulfureopectus;Sulphur-breasted Bushshrike;0.20446834
24.0;27.0;Cinclus pallasii;Pallaswasseramsel;0.14035383
39.0;42.0;Dicrurus adsimilis;Trauerdrongo;0.49781206
42.0;45.0;Dicrurus adsimilis;Trauerdrongo;0.16531901
48.0;51.0;Phoenicurus moussieri;Diademrotschwanz;0.56472397
60.0;63.0;Dicrurus adsimilis;Trauerdrongo;0.32988507
60.0;63.0;Phyllomyias burmeisteri;Burmeisterkleintyrann;0.1034713
63.0;66.0;Dicrurus adsimilis;Trauerdrongo;0.12177477
69.0;72.0;Dicrurus adsimilis;Trauerdrongo;0.24120894
69.0;72.0;Phyllomyias burmeisteri;Burmeisterkleintyrann;0.14926776
84.0;87.0;Riparia chinensis;Graukehl-Uferschwalbe;0.13066193
87.0;90.0;Tringa incana;Wanderwasserläufer;0.29393217
93.0;96.0;Riparia chinensis;Graukehl-Uferschwalbe;0.20919465
96.0;99.0;Eclectus roratus;Edelpapagei;0.35672444
96.0;99.0;Dicrurus adsimilis;Trauerdrongo;0.15679923
114.0;117.0;Furnarius minor;Kleintöpfer;0.13333207
123.0;126.0;Synallaxis brachyura;Graurücken-Dickichtschlüpfer;0.12138738
129.0;132.0;Riparia chinensis;Graukehl-Uferschwalbe;0.12164671
132.0;135.0;Troglodytes troglodytes;Zaunkönig;0.13889167

ghost commented 3 years ago

Have now installed the BirdNET-Lite from scratch without the German labels file. Getting this for the 'Amsel" (Blackbird): Amsel.mp3.zip

(base) ➜  BirdNET-Lite git:(main) python analyze.py --i ~/Music/SoundAndWaveFileExamples/Bird\ Sounds/Amsel.mp3 --lat 49.584 --lon 6.888 -week 11; cat result.csv
Init Plugin
Init Graph Optimizer
Init Kernel
usage: analyze.py [-h] [--i I] [--o O] [--lat LAT] [--lon LON] [--week WEEK] [--overlap OVERLAP] [--sensitivity SENSITIVITY] [--min_conf MIN_CONF] [--custom_list CUSTOM_LIST]
analyze.py: error: unrecognized arguments: -week 11
Start (s);End (s);Scientific name;Common name;Confidence
15.0;18.0;Junco hyemalis;Dark-eyed Junco;0.63864714
19.5;22.5;Poecile atricapillus;Black-capped Chickadee;0.28910682
21.0;24.0;Junco hyemalis;Dark-eyed Junco;0.2813258
21.0;24.0;Pipilo maculatus;Spotted Towhee;0.26970756
22.5;25.5;Junco hyemalis;Dark-eyed Junco;0.36897075
25.5;28.5;Poecile atricapillus;Black-capped Chickadee;0.36192033
28.5;31.5;Junco hyemalis;Dark-eyed Junco;0.54385275
34.5;37.5;Junco hyemalis;Dark-eyed Junco;0.6392924
43.5;46.5;Junco hyemalis;Dark-eyed Junco;0.395477
45.0;48.0;Junco hyemalis;Dark-eyed Junco;0.35142577
49.5;52.5;Junco hyemalis;Dark-eyed Junco;0.36010146
51.0;54.0;Junco hyemalis;Dark-eyed Junco;0.3235896
54.0;57.0;Junco hyemalis;Dark-eyed Junco;0.61683154
55.5;58.5;Junco hyemalis;Dark-eyed Junco;0.30236518
61.5;64.5;Junco hyemalis;Dark-eyed Junco;0.5145074
63.0;66.0;Junco hyemalis;Dark-eyed Junco;0.55427563
66.0;69.0;Poecile atricapillus;Black-capped Chickadee;0.7129054
67.5;70.5;Junco hyemalis;Dark-eyed Junco;0.4047741
69.0;72.0;Junco hyemalis;Dark-eyed Junco;0.70841223
70.5;73.5;Junco hyemalis;Dark-eyed Junco;0.25039598

In the BirdNET app i get the correct result. Don't undestand that.

euxoa commented 3 years ago

Well the app is different, don't know which model it has but it probably does some temporal pooling, and the results are better and more filtered than those from BirdNET-Lite.

Then your -week is unrecognised. Add a dash.

Third, none of the results raise above .7 in confidence, which implies that they are quite uncertain. How is the sound quality of your file? I have ended up grepping only results with confidence>.7, and even those are not all right. You need to do a bit of "temporal pooling" yourself, and familiarise yourself with the common errors the network makes.

But Junco hyemalis doesn't sound like a Blackbird at all, as far as I can tell. There can still be something wrong with your installation. Try with other files, and the examples provided with the software.

The most obvious thing to mess up with the translated files is to get async with the positions of the lines in the file. If you have an offset of one somewhere, the results are all garbage.

ghost commented 3 years ago

The app uses this model --> https://github.com/kahst/BirdNET-CoreML . Corrected the week and get:

(base) ➜  BirdNET-Lite git:(main) ✗ python analyze.py --i ~/Music/SoundAndWaveFileExamples/Bird\ Sounds/Amsel.mp3 --lat 49.584 --lon 6.888 --week 13 --overlap 1.5 --min_conf 0.7 --sensivity 1.25 ; cat result.csv 
Init Plugin
Init Graph Optimizer
Init Kernel
usage: analyze.py [-h] [--i I] [--o O] [--lat LAT] [--lon LON] [--week WEEK] [--overlap OVERLAP] [--sensitivity SENSITIVITY] [--min_conf MIN_CONF] [--custom_list CUSTOM_LIST]
analyze.py: error: unrecognized arguments: --sensivity 1.25
Start (s);End (s);Scientific name;Common name;Confidence
3.0;6.0;Turdus merula;Eurasian Blackbird;0.72955966
9.0;12.0;Turdus merula;Eurasian Blackbird;0.23384675
12.0;15.0;Turdus merula;Eurasian Blackbird;0.5658356
12.0;15.0;Turdus iliacus;Redwing;0.13565567
18.0;21.0;Turdus merula;Eurasian Blackbird;0.8005052
21.0;24.0;Turdus iliacus;Redwing;0.4076453
21.0;24.0;Turdus philomelos;Song Thrush;0.2655036
21.0;24.0;Periparus ater;Coal Tit;0.16680844
21.0;24.0;Hippolais icterina;Icterine Warbler;0.11638691
24.0;27.0;Numenius arquata;Eurasian Curlew;0.6342368
27.0;30.0;Turdus merula;Eurasian Blackbird;0.99626005
30.0;33.0;Certhia brachydactyla;Short-toed Treecreeper;0.78408766
33.0;36.0;Turdus merula;Eurasian Blackbird;0.7945151
36.0;39.0;Turdus merula;Eurasian Blackbird;0.51626205
39.0;42.0;Turdus merula;Eurasian Blackbird;0.89651906
42.0;45.0;Turdus merula;Eurasian Blackbird;0.11417044
45.0;48.0;Phoenicurus ochruros;Black Redstart;0.108772814
48.0;51.0;Turdus merula;Eurasian Blackbird;0.74833506
51.0;54.0;Hippolais icterina;Icterine Warbler;0.16020058

which is correct. :-) Was also able to reproduce this on the live system from Patrik. The main problem was the wrong labels.txt file. I was also able to reproduce other species.

patlevin commented 3 years ago

See #4 - I (hopefully) fixed the labels.

ghost commented 3 years ago

Can confirm that the labels are now fixed and work for all languages.

kahst / BirdNET-Lite

Classification Benchmark and false positives #11