[BUG] global indicator legend switched

[SanderDevisscher]

Describe the bug The icons on the global indicators page legend have been switched. The star should be (and is in the table) union concern species. While triangle should be (and is in the table) species with observations

To Reproduce Steps to reproduce the behavior:

1. Go to 'global indicators'
2. Click on 'legend'
3. See error

Expected behavior The star should be (and is in the table) union concern species. While triangle should be (and is in the table) species with observations

Screenshots

[SanderDevisscher]

also, the colors seem to be missing

[mvarewyck]

Should be fine now, see below

[mvarewyck]

As discussed: color is green if management data available, otherwise black

However, I found out that union list data contains a special white space at the end of "Oxyura jamaicensis". Therefore the star button is missing. I prefer to fix it in the data https://github.com/inbo/alien-species-portal/blob/sprint_v0.0.4/alienSpecies/inst/extdata/eu_concern_species.tsv#L38 Otherwise I will need to add an extra package dependency just for this single fix.

library(alienSpecies)
unionlistData <- loadTabularData(type = "unionlist")

trimws(unionlistData$scientificName)[37]
# [1] "Oxyura jamaicensis "

trimws(unionlistData$scientificName[37])
# [1] "Oxyura jamaicensis "

stringr::str_trim(unionlistData$scientificName[37])
# [1] "Oxyura jamaicensis"

[SanderDevisscher]

you still need to add the star for union concern species, correct?

[SanderDevisscher]

in data fix is ok!

alopochen should be a union list species as well though

[mvarewyck]

alopochen should be a union list species as well though

In the unionlist data it is spelled Alopochen aegyptiacus, while in the data it is Alopochen aegyptiaca. Another fix in the unionlist data? @SanderDevisscher This gives me the impression we should best check the whole union list..

[mvarewyck]

Half of the union list species is not present in the table. Do you see any abnormalities?

Browse[1]> nrow(unionlistData)
[1] 66
Browse[1]> unionlistData$scientificName[!unionlistData$scientificName %in% tableData$species]
 [1] "Alternanthera philoxeroides"      "Baccharis halimifolia"           
 [3] "Gunnera tinctoria"                "Heracleum persicum"              
 [5] "Heracleum sosnowskyi"             "Microstegium vimineum"           
 [7] "Pennisetum setaceum"              "Persicaria perfoliata"           
 [9] "Pueraria lobata"                  "Alopochen aegyptiacus"           
[11] "Eriocheir sinensis"               "Herpestes javanicus"             
[13] "Orconectes limosus"               "Orconectes virilis"              
[15] "Oxyura jamaicensis "              "Percottus glenii"                
[17] "Procambarus fallax f. virginalis" "Vespa velutina nigrithorax"      
[19] "Humulus scandens"                 "Andropogon virginicus"           
[21] "Arthurdendyus triangulatus"       "Cortaderia jubata"               
[23] "Acacia saligna"                   "Gymnocoronis spilanthoides"      
[25] "Triadica sebifera"                "Lepomis gibbosus"                
[27] "Lespedeza cuneata"                "Lygodium japonicum"              
[29] "Plotosus lineatus"                "Prosopis juliflora"              
[31] "Salvinia molesta"                 "Ehrharta calycina"  

[SanderDevisscher]

Welcome to the miraculous world of taxonomy where nothing is what it seems and nothing stays the same 😅 Most of these scientificNames are synonyms of the accepted scientificname.

Do you get the same level of dropouts if you use the taxonkey (backbone_taxonKey) in the unionlist ? Alopochen aegyptiacus & Lepomis gibbosus is an example which should be fixed by using the taxonKey. While species like Orconectes limosus will remain an issue since this is both a synonym in the textual sense as it is a synonym in the taxonomic sense of the accepted species Faxonius limosus.

[SanderDevisscher]

also, apparently the eu_concern_species.tsv you use is out of date. I've uploaded a newer version.

[SanderDevisscher]

This newer version should fix most of the issues concerning Orconectes limosus or should I say Faxonius limosus.

In the end few species on the unionlist will not be present in Belgium and thus be excluded from the cube. Keep in mind the cube is also outdated (see https://github.com/inbo/alien-species-portal/issues/50#issuecomment-1764602626) meaning some species that are currently present in belgium will remain absent because they were not present on gbif when the cube was made.

[mvarewyck]

Matching on the key works for 'Alopochen aegyptiaca' and 'Oxyura jamaicensis'. It doesn't for 'Vespa velutina'

> exotenData$nubKey[exotenData$species == "Vespa velutina"]
[1] 1311477 1311477 1311477
> unionlistData[49, ]
               scientificName  englishName  kingdom taxonKey
1: Vespa velutina nigrithorax Asian hornet Animalia  6247411

Species on the union list that are currently not matching with any record in the table

> nrow(unionlistData)
[1] 89
> unionlistData$scientificName[!unionlistData$taxonKey %in% exotenData$nubKey]
 [1] "Alternanthera philoxeroides" "Baccharis halimifolia"      
 [3] "Gunnera tinctoria"           "Heracleum persicum"         
 [5] "Heracleum sosnowskyi"        "Microstegium vimineum"      
 [7] "Pennisetum setaceum"         "Persicaria perfoliata"      
 [9] "Pueraria lobata"             "Eriocheir sinensis"         
[11] "Herpestes javanicus"         "Faxonius limosus"           
[13] "Faxonius virilis"            "Percottus glenii"           
[15] "Procambarus virginalis"      "Vespa velutina nigrithorax" 
[17] "Humulus scandens"            "Andropogon virginicus"      
[19] "Arthurdendyus triangulatus"  "Cardiospermum grandiflorum" 
[21] "Cortaderia jubata"           "Acacia saligna"             
[23] "Gymnocoronis spilanthoides"  "Triadica sebifera"          
[25] "Lepomis gibbosus"            "Lespedeza cuneata"          
[27] "Lygodium japonicum"          "Plotosus lineatus"          
[29] "Prosopis juliflora"          "Salvinia molesta"           
[31] "Ehrharta calycina"           "Ameiurus melas"             
[33] "Axis axis"                   "Callosciurus finlaysonii"   
[35] "Channa argus"                "Faxonius rusticus"          
[37] "Fundulus heteroclitus"       "Gambusia affinis"           
[39] "Gambusia holbrooki"          "Hakea sericea"              
[41] "Koenigia polystachya"        "Lampropeltis getula"        
[43] "Lampropeltis getula"         "Limnoperna fortunei"        
[45] "Morone americana"            "Rugulopteryx okamurae"      
[47] "Solenopsis geminata"         "Solenopsis invicta"         
[49] "Solenopsis richteri"         "Wasmannia auropunctata"    

[SanderDevisscher]

did you use the new eu concern list ?

[SanderDevisscher]

when using the taxonKey for Eriocheir sinensis from the new eu concern list (2225776) and filter the be_alientaxa_cube on this repo I get at least 1350 hits. Lepomis gibbosus yields similar results. While species like Vespa velutina remains problematic. The currently used taxonKey on the unionlist (6247411) is the one of a subspecies Vespa velutina nigrithorax while the cube was build with the taxonKey of the species (1311477).

I'll add some logic to pull the eu_concern_list from Trias and fix this type of issues before uploading it to the bucket.

[mvarewyck]

did you use the new eu concern list ?

~With the latest eu concern list, all species are matched.~

> unionlistData$scientificName[!unionlistData$taxonKey %in% exotenData$nubKey]
character(0)

Variable taxonKey was not in the unionlistData, which gave wrong output

[SanderDevisscher]

also for Vespa velutina ?

[mvarewyck]

No, still same list as reported here https://github.com/inbo/alien-species-portal/issues/57#issuecomment-1766159063

Wrt 'Eriocheir sinensis': It is in the data file 'data_input_checklist_indicators.tsv', but excluded because the first observation date is before 1950. All such records are excluded here in the code Is this no longer as intendend?

[SanderDevisscher]

I my opinion species introduced before 1950 should be included in all the lists but excluded or be a base load for cumulative indicators. does this make sense ?

@timadriaens what do you think?

[timadriaens]

I think that is a mistake as the indicators should show everything, e g. such as here? I suggest to ask @damianooldoni

[SanderDevisscher]

so drop 1950 filter altogether ?

[timadriaens]

what indicator is this about?

[SanderDevisscher]

what indicator is this about?

all of them

[timadriaens]

I think for all of the checklist based indicators (pathways, origin, introduction rates) we do NOT need this filter. I suspect this filter comes from the occurrence-based indicator preprocessing and of course it remains relevant there.

[damianooldoni]

Yes, we had the filtering while preprocessing occurrence based indicators, but not for checklist based ones.

[SanderDevisscher]

The 1950 filter should be removed in case of the checklist indicators. https://github.com/inbo/alien-species-portal/blob/e48584677113aec2226283d63f837876d5ab8e3d/alienSpecies/R/data_load.R#L202 But it can remain for the occurrence indicators https://github.com/inbo/alien-species-portal/blob/e48584677113aec2226283d63f837876d5ab8e3d/alienSpecies/R/data_load.R#L303

[mvarewyck]

The crucial ones you mentioned are now recognized in the data. These are the ones without a match still:

> unionlistData$scientificName[!unionlistData$taxonKey %in% exotenData$nubKey]
 [1] "Alternanthera philoxeroides" "Gunnera tinctoria"          
 [3] "Heracleum persicum"          "Heracleum sosnowskyi"       
 [5] "Microstegium vimineum"       "Pennisetum setaceum"        
 [7] "Persicaria perfoliata"       "Pueraria lobata"            
 [9] "Herpestes javanicus"         "Faxonius limosus"           
[11] "Faxonius virilis"            "Percottus glenii"           
[13] "Procambarus virginalis"      "Vespa velutina nigrithorax" 
[15] "Humulus scandens"            "Andropogon virginicus"      
[17] "Arthurdendyus triangulatus"  "Cardiospermum grandiflorum" 
[19] "Cortaderia jubata"           "Acacia saligna"             
[21] "Gymnocoronis spilanthoides"  "Triadica sebifera"          
[23] "Lespedeza cuneata"           "Lygodium japonicum"         
[25] "Plotosus lineatus"           "Prosopis juliflora"         
[27] "Salvinia molesta"            "Ehrharta calycina"          
[29] "Axis axis"                   "Callosciurus finlaysonii"   
[31] "Channa argus"                "Faxonius rusticus"          
[33] "Fundulus heteroclitus"       "Gambusia affinis"           
[35] "Gambusia holbrooki"          "Hakea sericea"              
[37] "Koenigia polystachya"        "Lampropeltis getula"        
[39] "Lampropeltis getula"         "Limnoperna fortunei"        
[41] "Morone americana"            "Rugulopteryx okamurae"      
[43] "Solenopsis geminata"         "Solenopsis invicta"         
[45] "Solenopsis richteri"         "Wasmannia auropunctata" 

[timadriaens]

Most of these do not occur in Belgium but there are a few where matching problzms could have occurred, such as Salvinia molesta (could be under auriculata), procambarus virginalis (Procambarus fallax), Vespa velutina nigrithorax (Vespa velutina), Koenigia (Persicaria wallichii). However, i thought we solved this several times already for riparias and Prius @damianooldoni

[SanderDevisscher]

I've crossreferenced with gbif and encountered a few issues:

Unionlist name	unionlist key	gbif key	remarks
Faxonius limosus	8909595	8909595	Missing from checklist & occurrence cube
Koenigia polystachya	8848208	8848208	Missing from checklist & occurrence cube
Lampropeltis getula	9799308	9799308	Missing from checklist & occurrence cube
Lampropeltis getula	8170369	8170369	Missing from checklist & occurrence cube
Procambarus virginalis	9442269	9442269	Missing from checklist & occurrence cube
Ehrharta calycina	7644403	7644403	HETEROTYPIC SYNONYM Not present in belgium
Heracleum persicum	3628745	3628745	No validated occurrences
Heracleum sosnowskyi	3642949	3642949	No validated occurrences
Persicaria perfoliata	4033648	4033648	No validated occurrences
Microstegium vimineum	5289808	5289808	No validated occurrences => 1 occurrence missing validation status in a life riparias dataset https://www.gbif.org/dataset/4ba6a4ac-bf1c-4631-a6d5-4f2d29d22e40
Salvinia molesta	5274863	5274861	Different species on checklist & in cube (Salvinia auriculata Aubl.)
Callosciurus finlaysonii	2437399	2437399	Not on checklist nor occurrence cube but at least 1 occurrence in belgium according to gbif (https://www.gbif.org/occurrence/2270403921)
Alternanthera philoxeroides	3084923	3084923	Not present in belgium
Andropogon virginicus	2706080	2706080	Not present in belgium
Arthurdendyus triangulatus	2502792	2502792	Not present in belgium
Channa argus	4284921	4284921	Not present in belgium
Cortaderia jubata	2704521	2704521	Not present in belgium
Faxonius rusticus	8979506	8979506	Not present in belgium
Faxonius virilis	8971201	8971201	Not present in belgium
Fundulus heteroclitus	5712056	5712056	Not present in belgium
Gambusia affinis	2350580	2350580	Not present in belgium
Gambusia holbrooki	2350570	2350570	Not present in belgium
Gymnocoronis spilanthoides	6063677	6063677	Not present in belgium
Hakea sericea	7287606	7287606	Not present in belgium
Humulus scandens	2984537	2984537	Not present in belgium
Lespedeza cuneata	8114276	8114276	Not present in belgium
Limnoperna fortunei	5855350	5855350	Not present in belgium
Lygodium japonicum	2650436	2650436	Not present in belgium
Morone americana	2394604	2394604	Not present in belgium
Plotosus lineatus	7965247	7965247	Not present in belgium
Prosopis juliflora	5358460	5358460	Not present in belgium
Rugulopteryx okamurae	5824863	5824863	Not present in belgium
Solenopsis geminata	5035187	5035187	Not present in belgium
Solenopsis invicta	5035230	5035230	Not present in belgium
Solenopsis richteri	5035017	5035017	Not present in belgium
Triadica sebifera	3054399	3054399	Not present in belgium
Wasmannia auropunctata	1315391	1315391	Not present in belgium
Axis axis	2440934	2440934	Not present in belgium according to gbif, is this correct ??
Percottus glenii	2390064	2390064	Not present in belgium according to gbif, is this correct ??
Gunnera tinctoria	2984306	2984306	Only recently (>2022) validated observations -> should be included with update of occurrence cube & checklist
Vespa velutina nigrithorax	6247411	1311477	subspp. On unionlist vs Species on checklist/cube
Pennisetum setaceum	2706134	5828232	SYNONYM -> Switch to species? Cenchrus setaceus (Forssk.) Morrone ?
Herpestes javanicus	8858565	2434271	Unionlist species deleted from gbif taxonomic backbone on 2018-02-01 -> switch to Herpestes javanicus (É.Geoffroy Saint-Hilaire, 1818) ? Not present in belgium
Pueraria lobata	9035634	9035634	Unionlist species deleted from gbif taxonomic backbone on 2021-03-03 -> switch to Pueraria lobata (Willd.) Ohwi ? Not present in belgium
Acacia saligna	8790482	2978552	Unionlist species is doubtfull -> deleted from gbif 2021-03-03 -> switch to Acacia saligna (Labill.) H.L.Wendl. ? Not present in belgium
Cardiospermum grandiflorum	9685025	3189935	Unionlist species is doubtfull -> switch to Cardiospermum grandiflorum Sw. ? Not present in belgium

[SanderDevisscher]

Most of these do not occur in Belgium but there are a few where matching problzms could have occurred, such as Salvinia molesta (could be under auriculata), procambarus virginalis (Procambarus fallax), Vespa velutina nigrithorax (Vespa velutina), Koenigia (Persicaria wallichii). However, i thought we solved this several times already for riparias and Prius @damianooldoni

Salvinia molesta is on the checklist as Salvinia auriculata. I'll add a fix like I'll do for Vespa velutina (see https://github.com/inbo/aspbo/issues/26).

Persicaria wallichii & Koenigia polystachya are both missing from both checklist & occ. cube.

As are Procambarus virginalis & Procambarus fallax

[timadriaens]

Persicaria wallichii & Koenigia polystachya are both missing from both checklist & occ. cube. --> maybe it is under Polygonum polystachyum?

Axis axis en Perccottus (mind the two "c's" in the spelling!) glenii do not occur in Belgium indeed

[SanderDevisscher]

Persicaria wallichii & Koenigia polystachya are both missing from both checklist & occ. cube. --> maybe it is under Polygonum polystachyum?

Polygonum polystachyum is also missing

Axis axis en Perccottus (mind the two "c's" in the spelling!) glenii do not occur in Belgium indeed

[timadriaens]

@damianooldoni is going to produce a new cube which should solve a number of these problems

[damianooldoni]

Yes, as @timadriaens said I will start making a new cube today. Still, I wanted check first the issues with the GRIIS checklist. I checked the very first taxa mentioned by @SanderDevisscher. Here my findings:

species	taxonKey (in GRIIS checklist)	GBIF backbone
Faxonius limosus	213214572	8909595
Koenigia polystachya	213214913	8848208
Lampropeltis getula	213212464	5224480
Procambarus virginalis	not present	9442269

Procambarus virginalis is not in the GRIIS checklist because:

1. it's not in the waarnemingen.be list (only P. acutus and P. clarkii) AND
2. it's not in the Inventory of alien macroinvertebrates in Flanders, Belgium. Maybe is there a way to update this checklist? If so, @timadriaens, please take contact with authors and write an issue on the dedicated repository: https://github.com/trias-project/alien-macroinvertebrates

I mentioned these two datasets because both contain another mentioned crayfish: Faxonius limosus.

I think @SanderDevisscher is still using an old version of the GRIIS checklist? Of course the cube will be created on the new and most updated version of the GRIIS checklist. I will also update the indicators as next step. @SanderDevisscher: within LIFE RIPARIAS project Tim and I published the Union Concern list on GBIF. So, maybe it's relevant to retrieve the information about such list directly from GBIF. This also allows you to automatize such step and get rid of csv's with such list spread among several repositories.

[timadriaens]

Indeed, the taxon on GRIIS Belgium should be Procambarus virginalis (9442269). It is normal since the taxonomy changed from P. fallax f. virginalis well after the checklist or macroinverts was published. Yet, is it then not better to include the original taxon (synonym, 8879526) in that source checklist @damianooldoni ? If so, I'll raise an issue to adapt that in the source.

[timadriaens]

Correction, it does not occur in this checklist of course as it was not yet present on the belgian territory then. But it should be in waarneminge.be checklist! I thought this wnm.be checklist was now included in GRIIS Belgium - maybe we should check here what happened with @peterdesmet ?

[peterdesmet]

What species do you expect in the waarnemingen.be checklist?

[timadriaens]

Procambarus virginalis (9442269), on wnm.be

[peterdesmet]

The species does occur on waarnemingen.be and is considered a species Procambarus virginalis : https://waarnemingen.be/species/561245/

For some reason, it didn't make it to the waarnemingen.be checklist however. Maybe because the rank changed from a form (not included in the checklist) to a species after the initial checklist was published. The only Procambarus that are included in the checklist are:

• Procambarus clarkii
• Procambarus acutus

[peterdesmet]

Correction, it seems to be (incorrectly) considered a variety in waarnemingen.be, and a child of Procambarus fallax (https://waarnemingen.be/species/561244/) which is also not included in the checklist.

[timadriaens]

OK, so can we change that? It's a Union List species...

[SanderDevisscher]

@mvarewyck do we currently use the entry_into_force date from the eu_concern_species.tsv?

[SanderDevisscher]

This issue is waiting for updates of the GRIIS checklist & alien species cube to be solved.

[mvarewyck]

@mvarewyck do we currently use the entry_into_force date from the eu_concern_species.tsv?

@SanderDevisscher No, we only read the following variables "checklist_scientificName", "english_name", "checklist_kingdom", "backbone_taxonKey"

[peterdesmet]

@timadriaens, concerning the missing "Procambarus virginalis" in the waarnemingen.be list: I have created an issue for that in data-publication-natuurpunt. The list will likely be updated in January 2024.