Insights - display unknown species in a different way

[naluinui]

Background

Original from https://rfcx.slack.com/archives/C02DJUZLNCE/p1711723120125409 @krisharmon

Stumbled across sp1 in Insights. Have we always included unknowns? Seems a bit funky, but maybe there are legit reasons why it makes sense to? The project in question is here.

The question is really around whether or not unknown species should or should not be displayed on the Insights page. If we think they should, my next question is if it makes sense to treat them differently in some way. Just wondering out loud, not sure if this has been discussed and decided previously.

Expect result

• [x] Clear the auto generate data (from taxon_species_wiki table) for species sp1-sp50 from the database
• [x] Don't sync the description data from wikipedia for those species anymore (to prevent the text to come back in the future)

• [ ] All the unknown species shouldn't have irrelevant description, and should be "No information found" as screenshot below

  Screenshot from https://arbimon.org/p/tricolored-blackbird-breeding-phenology/insights/spotlight/sp22?guest=1

[carlybatist]

Plan: Hide sp1-sp50 from Insights pages @naluinui

[naluinui]

There are 473 species used across 122 projects (37 published, 61 listed, 17 unlisted, 7 hidden)

Breakdown of effected projects

count	slug
1	1485-juliana-project
1	1662-aya-project
3	a2-workshop
38	acoustic-community-phenology-in-colombian-andes
2	actividad-ac-stica-anuros
1	african-parks-australia-testing
1	aguas-de-santa-barbarba
1	amphibians-peccary-wallows
1	andean-soundscape
2	anh-putumayo
1	apuane-howlscape
1	asri-bioaccoustic-trial-for-survey
1	audiomoth-rnnes
1	audiomoth-tests-xuleta
1	bacalar
2	bachelorarbeit
36	bci-panama-2018
2	bestaeuber
4	bioacustica-anuros-tena
1	biosoundscape
3	biosoundscape-urban
1	bompu-pilot
2	canande-rfcx-fundaci-n-jocotoco
1	cenicana-humboldt
1	cervus-elaphus-corsicanus
1	chainsaw-project-koronia
1	chirimoya
8	coal-road-hutan-harapan-p4f
44	comoros-birds
1	congo-chainsaw-project
1	congo-data
25	corredor-jaguar
1	cws-india-elephants
1	dcase-2023
1	deep-project-sumatra
1	destinos-awake
1	detection-of-the-tokay-gecko-in-martinique
1	Drakensberg
1	dunbar
1	eij-in-eastern-north-carolina
1	essai1
1	explosive-anuran-breeding-in-french-guiana
1	Faunal-Response-Fire-Amazon
1	freq-entaci-pnsll
1	fundaci-n-jocotoco-sigchos
1	greece-aoos-gorge
6	gryllotalpa-marismortui
1	hoverfliestest
2	hulu-batang-hari-sumatra
1	iesa-phj-lbb-honduras
37	ivohibory-forest
1	jama-coaque-reserve-ecuador-caip
1	jgi-rfcx-gombe-tz
2	joeri-s-project
1	kenya-trees-forest-gardens
2	kinabatangan-malaysia-sabah
25	la-estrella
9	lima-bats
2	madagascar-lemur-data
1	mdd-bats
3	monitoreo-acustico-uraba
1	monte-arcosu-wwf
5	mount-kenya-landscape
1	my-first-project
1	nogomonitortest
7	ocean-soundscapes
3	ondas-marine-communities-of-central-america
4	padr-es-de-biodiversidade-na-mata-atl-ntica
2	paisagem-ac-stica-pnmni
2	paisaje-marino-1-0
1	perm-stations
2	peruvian-tern-and-rn-paracas-soundscape
1	piha-1
5	pipeline-and-gamboa-roads-2022
3	pnc-putumayo
2	PozoNancy
1	project-kalamies
2	proyecto-farallones-2021-2022
1	pruebaarbimon
2	prueba-ranas-de-madre-de-dios
1	puerto-rico-island-wide
2	red-squirrel-monitoring
1	reed-sounds-of-lake-neusiedl
23	rewilding-madagascar
1	rfcx-external-templates
1	rfcx-guardians-madre-de-dios-peru
1	rhodope-project
1	romania-carpathia-mossy-earth-guardians
1	sandybar
1	santa-barbara
2	serc-river-herring
5	shiripuno-march-2023-weeks-expedition
1	skripsweet-tata
3	smithsonian-wandari-project
1	soundscape-mata-choupal
43	soundscape-of-deforestation
3	sounds-of-atlantic-forest
2	sunyesf-scorecard-test
1	sustainable-cattle-ranching-in-colombia
1	temb-reserve-brazil-guardians
1	temb-reserve-brazil-species
2	test-arbimon-safari
1	teste-1
1	teste_fibria
1	testingsg
1	test-me
2	tfm-acustica
2	the-amphibians-of-rio-muni-equatorial-guinea
1	trabajo-de-grado
4	tricolored-blackbird-breeding-phenology
1	uganda-bwindi
3	ukan-pond-project
1	upeace-acoustic-monitoring
3	usfws-carite-warbler
1	usfws-puerto-rican-parrot-rio-abajo
1	volcanoes-national-park-rwanda-and-surroundings
1	vozes-da-conserva-o-aves-amea-adas-do-cear
1	watchout-kopacki-rit-nature-park-croatia
1	wild-bees
1	wildbees
3	wwf-peru
1	yosemite-toad

[naluinui]

Instead of hiding those species from the Insights. Can we just clear all the auto generate data for sp1-sp50 to be similar to this?

Screenshot from https://arbimon.org/p/tricolored-blackbird-breeding-phenology/insights/spotlight/sp22?guest=1

This will only take us less than 30 mins to clear those auto generated data from the database VS at least 2 days or a whole sprint of work to hide those species from the Insights.

[carlybatist]

I'm fine with your solution here @naluinui

[naluinui]

ok then. I think we can get it in this sprint.

[grindarius]

@carlybatist will these list continue to grow in the future? like could there be plan to add sp51 in in the future?

[grindarius]

From this SQL

select
    taxon_species_wiki.*,
    taxon_species.slug as slug
from taxon_species_wiki 
right join taxon_species on taxon_species_wiki.taxon_species_id = taxon_species.id
where taxon_species.slug ~ '^sp[0-9]+$';

Only sp1 to sp7 that have the value, I will be resetting it on testing env to try it.

[naluinui]

@grindarius I don't think we'll be adding more of these species in the future. Yes just resetting description for 7 of those species is fine for now. Thanks

[grindarius]

I will work on some fix on the daily sync script so that these species does not occur again using a regexp match on the script side, I think I won't be doing any regexp matches on the database side because it looks to be expensive where we can have expensive logic in javascript.

The regex101 link if you want to see it https://regex101.com/r/tsc0pT/1

[grindarius]

I have checked inside the database and found that the sp based species is still being synced over because the regexp contains a g flag. I have removed it and also cleared all wiki related data out of all 3 databases.

[koonchaya]

Staging result

[grindarius]

@koonchaya I will keep an eye on staging daily sync and will report back whether the script ignores these sp species correctly or not.