Update aggregated butterfly species on eBMS web and app -

CrisSevilleja commented 6 months ago

Hello. I have been checking the aggregated butterfly species and I am missing some species on the app that I used before and they are not present at the moment (like Pieris rapae/napi). checking on ButterflyCount and the website, I see differences, there are not the same aggregated species in both systems when they should be equal (ButterflyCount has less aggregated species).

Also, I would like to update them including more new aggregated species that would be useful for users. I have attached you a document with the aggregated species present at the moment on the app, the website and the ones I would like to include. Thank you in advance. Aggregated species list.docx

chrisvanswaay commented 6 months ago

I noticed this with my first white of the year last weekend, but I thought I had mixed up platforms when I tried to enter Pieris napi/rapae (as it was too fast to be sure which of the whites it was). Good to add these aggregates back again.

DavidRoy commented 6 months ago

This might be a consequence of new functionality for species list to be restricted by your country setting. I suspect the aggregates are not tagged with each country they are present. The quick fix is to set species list = global in the app settings but we’ll also assign aggregates as present in countries so this doesn’t catch out other recorders.

chrisvanswaay commented 6 months ago

Is there a way we can add the aggregates to the country table?

chrisvanswaay commented 6 months ago

(although it could also be done by the programmers, as you can add an aggregate in each country where both species are already listed, shouldn't have to be very difficult in sql).

CrisSevilleja commented 5 months ago

The normal aggregates we were used to register for Europe are not in the global species list,. Meanwhile this is solved for species list restricted to countries, I would include the most important aggregates for the global list. thanks

johnvanbreda commented 1 month ago

@DavidRoy how is the global species list tagged? I'm not 100% sure what is needed for this issue having done most of #678. There are aggregates in the document linked in the original post such as Colias crocea/myrmidone which are not in list 251 (eBMS Butterflies) - do they need to be added and linked to any country where both species are present?

DavidRoy commented 1 month ago

@johnvanbreda the global list is everything in list 251 please can you add the missing aggregates from Cristina's document to list 251.

if sql is straightforward, yes you can add the aggregates to each country where both species are present

johnvanbreda commented 1 month ago

Will do. I will have a look at the SQL - it may be complex though.

Another question, for the aggregate Lampides boeticus/pirithous, we don't currently have L. pirithous in the database so shall I add it and, if so, which countries should it be assigned to?

DavidRoy commented 1 month ago

It should be Lepides boeticus/Leptotes pirithouse

chrisvanswaay commented 1 month ago

It's Lampides boeticus and Leptotes pirithous. An aggregate is difficult as they are two different genera, but of course still possible.

DavidRoy commented 1 month ago

sorry, yes my typo! We have other examples of aggregates across genera Lampides boeticus/Leptotes pirithouse as the aggregate name

chrisvanswaay commented 1 month ago

No problem. Just to be sure: the aggregate name should be without an e at the end, so Lampides boeticus/Leptotes pirithous.

chrisvanswaay commented 1 month ago

One of the advantages of always using the scientific name (in stead of the Dutch name) in the field and everywhere is that you spot such typos immediately.

johnvanbreda commented 1 month ago

Thanks all. Another question - do we need both Pieris napi/rapae and Pieris rapae/napi? If so can they be considered synonymous? Same question for Thymelicus sylvestris/lineola and Thymelicus lineola/sylvestris?

CrisSevilleja commented 1 month ago

Indeed it is synonymous. I placed it in a different order because if you want to use the aggregate Pieris napi/rapae, you cannot find it writing rapae, you have to write Pieris napi and then it appears. The same for Thymelicus sylvestris/lineola.

And Lampides boeticus/Leptotes pirithous are not from the same genus but can be confused quite a lot. Thank you for updating this list of aggregate species, quite useful for the volunteers.

larspett commented 1 month ago

Do they exist at the moment, all (?) butterfly aggregates seem gone from the Swedish version.

12 aug. 2024 kl. 12:12 skrev John van Breda @.***>:

Thanks all. Another question - do we need both Pieris napi/rapae and Pieris rapae/napi? If so can they be considered synonymous?

— Reply to this email directly, view it on GitHubhttps://github.com/BiologicalRecordsCentre/ABLE/issues/665#issuecomment-2283578120, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEVQXZH2NICSSLPPS6EMHO3ZRCDBLAVCNFSM6AAAAABFJGB4L6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBTGU3TQMJSGA. You are receiving this because you are subscribed to this thread.Message ID: @.***>

chrisvanswaay commented 1 month ago

Actually they were also there in Dutch, but have gone now, I can only choose Pieris if I see a fast white flying by.

johnvanbreda commented 1 month ago

@larspett I am not sure why you are not seeing any aggregates in the Swedish version - as an example I've checked Pieris napi/rapae and it is flagged as present in Sweden.

johnvanbreda commented 1 month ago

I've written an SQL script that works out which countries to link the combined aggregates to. Here is the logic used: 1) Find all taxon names that are of the form "genus species/species", "genus species/species/species" or "genus species/genus species", i.e. 2 or 3 species aggregate names, or 2 genus aggregate names. Other name forms are ignored. 2) Calculate the 2 or 3 species names that these combined names break down into. 3) Find countries which are linked to all the separate aggregate names, but are not linked to the combined aggregated name. These will be linked to the combined aggregate name.

As a result of this script I've produced the attached file which lists all the aggregate names along with the countries they will be linked to. Please can someone confirm that this looks reasonable before I actually attach link the countries to the names?

data-ebms-country-aggregates.csv

chrisvanswaay commented 1 month ago

Thanks @johnvanbreda , good idea.

I once sent a species/country list, but can resend if needed. No aggregate species are needed in countries where only one species occurs. E.g. B napaea does not occur in many countries (only Alps and mountains in Scandinavia), so no need for an aggregate in other countries.
Not sure where B napaea/aquilonaris is a problem, as far as I know they completely exclude each other, but I might not know all locations.
However B pales and napeae can occur mixed and can be hard to tell apart. That aggregate would be an improvement.
You also included aggregates where the species don't occur, e.g. Brenthis daphne/ino,"GB". Both species do not occur in GB.
H semele/senthes/volgensis. I think semele and the good aggregate senthes/volgensis never overlap.
Leptidea: in most of Europe you only have sinapis and juvernica, only in F and E reali also occurs.
L juvernica does not occur in GB, so no need for an aggregate there. This will also apply to some other places (see species/country file).
P daplidice/edusa: I think this is only a problem in CH and I, in the rest of Europe it is either daplidice or edusa, no need for an aggregate.
I think P malvae/malvoides is missing, in the border zone (so F, CH, A and I) you simply can't tell. In the rest of Europe is it only one of the species.
Also M athalia/celadussa is missing, in the contact zone (comparable to the Pyrgus malvae/malvoides) you cannot tell.

chrisvanswaay commented 1 month ago

Here is the species country list again. butterfly_species_country_list_2023.csv

NA for status means the species does not occur there, so you can also delete those lines. Legend:

You can use only these statuses (as I did for the RL): status=="P" | status=="M" | status=="I" | status=="Ex" | status=="P?"

larspett commented 1 month ago

Those butterfly aggregates disappeared from the app months ago but I thought it was temporary but they are still gone. They may still be present in the backend but are definitely missing from the app.

Also, there are other strange aggregate behaviours in the app. For moths, the aggregate Mesapamea secalis/secalella always becomes “unknown” when identified using an image even though the classifier deals with it correctly in ObsIdentify. If you update the name manually to that above in the app, that works but when you click ok then the entry disappears from the species list due to be uploaded. The same happens over and over again if there are more entries of this species in the same visit including if it is added manually without a picture. If you upload the species list from that visit, those entries are visible on the website. If you look at uploaded visits in the app, they are not there. If you would do this with the Swedish species name, Mindre vitaxfly/Vitaxfly, they will be visible in the list, both after tagging images and after adding them as a separate entry, those observations will be uploaded and they will be visible in the list of uploaded observations in the app list.

12 aug. 2024 kl. 16:26 skrev John van Breda @.***>:

@larspetthttps://github.com/larspett I am not sure why you are not seeing any aggregates in the Swedish version - as an example I've checked Pieris napi/rapae and it is flagged as present in Sweden.

— Reply to this email directly, view it on GitHubhttps://github.com/BiologicalRecordsCentre/ABLE/issues/665#issuecomment-2284135137, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEVQXZHF2U5E6PN5MUQMZMDZRDA23AVCNFSM6AAAAABFJGB4L6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBUGEZTKMJTG4. You are receiving this because you were mentioned.Message ID: @.***>

chrisvanswaay commented 1 month ago

I can confirm they have disappeared from my app in Dutch too since months. I could always choose for Pieris napi/rapae, but now that is not possible anymore and I have to move to the genus Pieris for a fast flying white.

chrisvanswaay commented 1 month ago

As addition to the aggregates: in France Boloria pales/napaea/graeca can be hard to distinguish when you do a 15min count. I think the French Alps are the only place where this could be a problem, maybe the Italian Alps as well.

CrisSevilleja commented 1 month ago

I've written an SQL script that works out which countries to link the combined aggregates to. Here is the logic used:

Find all taxon names that are of the form "genus species/species", "genus species/species/species" or "genus species/genus species", i.e. 2 or 3 species aggregate names, or 2 genus aggregate names. Other name forms are ignored.

Calculate the 2 or 3 species names that these combined names break down into.

Find countries which are linked to all the separate aggregate names, but are not linked to the combined aggregated name. These will be linked to the combined aggregate name.

As a result of this script I've produced the attached file which lists all the aggregate names along with the countries they will be linked to. Please can someone confirm that this looks reasonable before I actually attach link the countries to the names?

data-ebms-country-aggregates.csv

I can send this list to the coordinators to check their own countries and tell us which aggregates make sense and which are missing. However, I don't think I will receive many answers soon in the middle of the holiday season.

johnvanbreda commented 1 month ago

@DavidRoy @kazlauskis I'm not sure why the aggregates are not appearing in the Dutch/Swedish app versions (perhaps others), as they are linked in the database OK from what I can see. Perhaps this should be moved to a separate issue though?

Also @DavidRoy a full review of all the links between countries and the list of taxa seems like a change in scope of this issue - shall I wait for the review or proceed and apply the automatically calculated links between aggregates and countries (i.e. link them if all the constituent species are present in the country)?

chrisvanswaay commented 1 month ago

Maybe this is somehow linked to https://github.com/BiologicalRecordsCentre/ABLE/issues/709, where we can also see the Dutch names on the website (so they are in the database), but NOT when entering bumblebees in the field on the app.

kazlauskis commented 4 weeks ago

@johnvanbreda I have re-run the reports/projects/ebms/ebms_app_species_list.xml?taxon_list_id=251 report and there seem to be some new aggregates, but looking at the data-ebms-country-aggregates.csv I don't see all of them. For example:

Pieris rapae/brassicae is present in the excel sheet with a number of country codes - assuming country_code column means it's present there. The report correctly returns this aggregate as a latin name, but it doesn't have any presence attributes attached, while Pieris rapae/napi seems to be correct.

  {
    "taxa_taxon_list_id": "636546",
    "preferred_taxa_taxon_list_id": "473307",
    "taxon_meaning_id": "233270",
    "taxon_id": "589035",
    "taxon": "Pieris rapae/napi",
    "preferred_taxon": "Pieris napi/rapae",
    "attributes": "AD=P | AL=P | AT=P | BA=P | BE=P | BG=P | BY=P | CH=P | CY=P | CZ=P | DE=P | DK=P | EE=P | ES=P | ES: CA=P | Europe=P | FI=P | FR=P | GB=P | GR=P | HR=P | HU=P | IE=P | IT=P | LI=P | LT=P | LU=P | LV=P | MD=P | ME=P | MK=P | MT=P | NL=P | NO=P | PL=P | PT=P | RO=P | RS=P | SE=P | SI=P | SK=P | TR=P | UA=P",
    "language_iso": "lat",
    "taxon_group": 251
  },
{
    "taxa_taxon_list_id": "636544",
    "preferred_taxa_taxon_list_id": "636544",
    "taxon_meaning_id": "286427",
    "taxon_id": "589033",
    "external_key": "eBMS876",
    "taxon": "Pieris rapae/brassicae",
    "preferred_taxon": "Pieris rapae/brassicae",
    "language_iso": "lat",
    "taxon_group": 251
  }

larspett commented 4 weeks ago

None of the aggregates are visible in the app in Sweden at the moment/Lars

4 sep. 2024 kl. 14:01 skrev Karolis Kazlauskis @.***>:

@johnvanbredahttps://github.com/johnvanbreda I have re-run the reports/projects/ebms/ebms_app_species_list.xml?taxon_list_id=251 report and there seem to be some new aggregates, but looking at the data-ebms-country-aggregates.csvhttps://github.com/user-attachments/files/16588209/data-ebms-country-aggregates.csv I don't see all of them. For example:

Pieris rapae/brassicae is present in the excel sheet with a number of country codes - assuming country_code column means it's present there. The report correctly returns this aggregate as a latin name, but it doesn't have any presence attributes attached, while Pieris rapae/napi seems to be correct.

{ "taxa_taxon_list_id": "636546", "preferred_taxa_taxon_list_id": "473307", "taxon_meaning_id": "233270", "taxon_id": "589035", "taxon": "Pieris rapae/napi", "preferred_taxon": "Pieris napi/rapae", "attributes": "AD=P | AL=P | AT=P | BA=P | BE=P | BG=P | BY=P | CH=P | CY=P | CZ=P | DE=P | DK=P | EE=P | ES=P | ES: CA=P | Europe=P | FI=P | FR=P | GB=P | GR=P | HR=P | HU=P | IE=P | IT=P | LI=P | LT=P | LU=P | LV=P | MD=P | ME=P | MK=P | MT=P | NL=P | NO=P | PL=P | PT=P | RO=P | RS=P | SE=P | SI=P | SK=P | TR=P | UA=P", "language_iso": "lat", "taxon_group": 251 }, { "taxa_taxon_list_id": "636544", "preferred_taxa_taxon_list_id": "636544", "taxon_meaning_id": "286427", "taxon_id": "589033", "external_key": "eBMS876", "taxon": "Pieris rapae/brassicae", "preferred_taxon": "Pieris rapae/brassicae", "language_iso": "lat", "taxon_group": 251 }

— Reply to this email directly, view it on GitHubhttps://github.com/BiologicalRecordsCentre/ABLE/issues/665#issuecomment-2328781380, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEVQXZDQE56QVGBU2U3Y7JDZU3ZCJAVCNFSM6AAAAABFJGB4L6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRYG44DCMZYGA. You are receiving this because you were mentioned.Message ID: @.***>

chrisvanswaay commented 4 weeks ago

I can also confirm that aggregates (as Pieris napi/rapae) are not in the Dutch list on the app. As a result we now have to use the genus (Pieris) for such occasions, but almost always the doubt is between rapae and napi. That was what was good about the aggregate.

kazlauskis commented 4 weeks ago

This is still work in progress and I'll update the app with the latest species list once it is clear why the report doesn't return all the aggregates.

BiologicalRecordsCentre / ABLE

Update aggregated butterfly species on eBMS web and app - #665