Production Training Cycle for 2.12

[alexshepard]

• [x] pick a data export and bless as release version
• [x] do same with taxon range evals
• [x] download export and taxon range files
• [x] delete unneeded photos from previous training run
• [x] download validation photos
• [x] download training photos
• [x] make cleaned train and validation exports
• [x] train CV model
• [x] train geo model
• [x] make thresholds for geomodel
• [x] evaluate geomodel & thresholds using taxon range eval files
• [x] evaluate models against previous release
• [x] share models, thresholds, eval metric CSV artifacts to smilax NFS drive
• [x] release to staging
• [x] discuss with team approval to release
• [x] release publicly
• [x] write & publish announcement blog post

[alexshepard]

export 20240218060015 export chosen for model version 2.12

[alexshepard]

86,861 taxa, 47,697,109 train images

[alexshepard]

training on second-vm

[alexshepard]

note - need to lower wandb reporting intervals, we seem to be reporting too often. I changed this interval to every 100 steps from every 5 steps.

[alexshepard]

geomodel reported evaluation statistics: PR-AUC: 0.707 Precision: 0.65 Recall: 0.703 F1: 0.628

[alexshepard]

a dashboard for the CV training run is here: https://wandb.ai/alexsh/iNat%20CV/reports/2-12-Training-Run--Vmlldzo2OTgxNDIy

note - this is currently on my personal wandb account, which I've been using to evaluate wandb. Now that I've decided to use it going forward, I'll make a work wandb account and do all future training runs under a work/inat org account.

[alexshepard]

train stalled - rebooted second-vm and restarted after epoch 12

[alexshepard]

cv run finished this morning, all model weights and artifacts are up on smilax as transfer_2_12_20240218

[alexshepard]

blog post up here https://www.inaturalist.org/blog/91824-new-computer-vision-model-v2-12-with-1-983-new-taxa