drivendataorg / zamba

A Python package for identifying 42 kinds of animals, training custom models, and estimating distance from camera trap videos
https://zamba.drivendata.org/docs/stable/
MIT License
118 stars 27 forks source link

Release time distributed model trained on ~250k videos #226

Closed ejm714 closed 2 years ago

ejm714 commented 2 years ago

Replaces time distributed model trained on 15k videos with one trained on 250k videos. The same (improved) frame selection is used for both).

Key decision

This removes the 15k model so there is only one "time distributed" model available as an official model.

Rationale

As seen in the val_metrics.json, the new model does not have improved performance on every species judged by the F1 score. However, the new model has been evaluated on 30k instead of 3k videos and includes a greater diversity of videos given the inclusion of the Chimp & See data. From exploration into calibration and gains curves, it appears that there is still signal for the species with low or zero F1 score, it is just that probabilities are of lower confidence (e.g. highest probabilities still contain an animal, but those probabilities are less than 0.5). The old model was also trained on a much more balanced dataset (blanks at 15%) compared to the new model (blanks account for around 40%). The new model does significantly better on blank performance (f1 of 0.83 vs. 0.54) while also maintaining high performance on other species.

Closes https://github.com/drivendataorg/pjmf-zamba/issues/129

netlify[bot] commented 2 years ago

Deploy Preview for silly-keller-664934 ready!

Name Link
Latest commit e33df8d5248ebd2369314e18fc589e7d994429a4
Latest deploy log https://app.netlify.com/sites/silly-keller-664934/deploys/6321fe290a10c700091c887b
Deploy Preview https://deploy-preview-226--silly-keller-664934.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

github-actions[bot] commented 2 years ago

πŸš€ Deployed on https://deploy-preview-226--silly-keller-664934.netlify.app

codecov-commenter commented 2 years ago

Codecov Report

Merging #226 (e33df8d) into master (570f9cc) will not change coverage. The diff coverage is n/a.

@@          Coverage Diff           @@
##           master    #226   +/-   ##
======================================
  Coverage    87.0%   87.0%           
======================================
  Files          29      29           
  Lines        1937    1937           
======================================
  Hits         1686    1686           
  Misses        251     251           
Impacted Files Coverage Ξ”
zamba/models/config.py 96.7% <ΓΈ> (ΓΈ)
pjbull commented 2 years ago

πŸš€ !!!