Open yroskov opened 2 years ago
Name and legal issues. “LepIndex”, identical to NHM dataset, and a full name “Global Lepidoptera Index” vs NHM “The Global Lepidoptera Names Index” (https://www.nhm.ac.uk/our-science/data/lepindex/lepindex/). Both titles are almost identical for users. For sure, it will create a confusion in the information space. People will mess LepIndex @ NHS with LepIndex @ TaxonWorks. Appropriate explanation of a new life for LepIndex data should appear at NHM website before we include LepIndex @TaxonWorks in CoL. 2022-05-06: "TaxonWorks Lepidoptera"
Version. “2022-04-39”, i.e. in a style of date. Users will understand this Users will understand this (as shown in the CoL/CLB interface) as two dates together “2022-04-39 / 2022-04-30”, the first with typo. Would you please consider to change a style of version, for example as ver. “39, Apr 2022”, or as “2022-39” (if “39” is a linear iteration, not a typo). 2022-05-06: "2022-04-30 / 2022-04-30"
Authors & editors. No names so far. Any delimitation between Creators, Editors & Contributors? Names from original website (George Beccaloni, Malcolm Scoble, Ian Kitching, Thomas Simonsen, Gaden Robinson, Brian Pitkin, Adrian Hine & Chris Lyal). Definitely, we need names before we release dataset as a part of CoL.
Publisher. Do you consider Species Files Group to be the publisher of the dataset?
Abstract. I would suggest to add time intervals in two phrases to make steps transparent. “LepIndex [which version?] was subsequently imported into TaxonWorks [in 20??-20??] and is now maintained as an online curated resource”. “Significant updates have been made [in 20??-20??] to some families…”. ( It’s important to know, which version of LepIndex was imported in TaxonWorks. NHM website says “Database last updated January 2018”, CoL used a version 12.3 of Jan 2012. Nice to know, which data were imported in TW and when).
[x] Classification: used ranks: order (no authorship) - family (no authorship) - subfamily (no authorship) e.g. Adelidae tribe - genus (all genera with authors!) - species - subspecies
[ ] Big set of "empty" genera in the root of Lepidoptera
TASKS (2022-05-06)
https://www.checklistbank.org/catalogue/3/dataset/55434/tasks
@dhobern, what to do with 5 "pairs" of identical names (ACC-ACC species, same authors): https://www.checklistbank.org/catalogue/3/dataset/55434/duplicates?authorshipDifferent=false&category=binomial&limit=500&minSize=2&mode=STRICT&offset=0&status=accepted&withDecision=false Capua reclina Razowski, 1978 Capua reclina Razowski, 1978 Coleophora parasalmani Oudejans, 1971 Coleophora parasalmani Oudejans, 1971 Paranthrene poecilocephala Diakonoff, 1967 Paranthrene poecilocephala Diakonoff, 1967 Plebejus casaicus Tutt Plebejus casaicus Tutt Plebejus casaicus Tutt Ypthima chinensis Leech Ypthima chinensis Leech
@dhobern, Report with 471 identical names (ACC-SYN_spp_diffAcc_sameAuth) where one is an accepted species, another is a synonym to different accepted species, may help to catch "ghost names from the past": https://www.checklistbank.org/catalogue/3/dataset/55434/duplicates?acceptedDifferent=true&authorshipDifferent=false&category=binomial&limit=500&minSize=2&mode=STRICT&offset=0&status=accepted&status=synonym
Examples:
3 genera Acanthobrahmaea, Brahmaeops, Brahmidia in the family Brahmaeidae
https://www.checklistbank.org/dataset/55434/classification?taxonKey=234098
Subfamily Hibrildinae in the family Eupterotidae
https://www.checklistbank.org/dataset/55434/classification?taxonKey=234110
also:
[Hoplophanes] lithocolleta Turner, 1916
See family Incurvariidae with 9 unplaced species:
Other examples:
[GENUS NOT SPECIFIED] brunnea Schaus, 1901
[GENUS NOT SPECIFIED] poltrona Schaus, 1910
Record [GENUS NOT SPECIFIED] (supergen. Ganisa) in the family Eupterotidae:
https://www.checklistbank.org/dataset/55434/classification?taxonKey=235107
Other examples:
Alaria Duncan [& Westwood] , 1841 Aricia R.L. [Reichenbach] , 1817 Chloridea Duncan [& Westwood] , 1841
Acalyptris paradividua imkeviciute & Stonis, 2009
Synonyms with auctorum and nec-see below (yr: possibly Misapplied Names - check with Donald before final sync): Alucita auctorum Hypolamprus auctorum Ligia auctorum Cibyra (Philaenia) auctorum Anops phaedrus (Boisduval, 1836 nec Drury, 1773) Antithesia tripunctana (Stephens, 1834 nec Frölich, 1828)
Full list of Unparsable Authorship https://www.checklistbank.org/dataset/55434/names?facet=rank&facet=issue&facet=status&facet=nomStatus&facet=nameType&facet=field&facet=authorship&facet=authorshipYear&facet=extinct&facet=environment&facet=origin&issue=unparsable%20authorship&limit=1000&offset=0&reverse=false
ISSUES assessed 2022-05-26,27
Rank Invalid, 5 accepted species like this: [GENUS NOT SPECIFIED] (supergen. Apatetris) with rank "other" https://www.checklistbank.org/catalogue/3/dataset/55434/workbench?facet=rank&facet=issue&facet=status&facet=nomStatus&facet=nameType&facet=field&facet=authorship&facet=authorshipYear&facet=extinct&facet=environment&facet=origin&issue=rank%20invalid&limit=100&offset=0 = BLOCKED
Parent Species Missing, 38 accepted subspecies with missing parent species. https://www.checklistbank.org/catalogue/3/dataset/55434/workbench?facet=rank&facet=issue&facet=status&facet=nomStatus&facet=nameType&facet=field&facet=authorship&facet=authorshipYear&facet=extinct&facet=environment&facet=origin&issue=parent%20species%20missing&limit=100&offset=0 = NO ACTIONS IN CLB (for @dhobern to fix in TW)
Subspecies Assigned, 151. At least some of these names, CLB incorrectly parsed species names with two-word epithet as subspecies. Examples, Agrias pseudo lesoudieri Le Moult, 1926; Agrias semi rileyi Michener; Cosmopteryx sancti vincentii Walsingham, 1892 https://www.checklistbank.org/catalogue/3/dataset/55434/workbench?facet=rank&facet=issue&facet=status&facet=nomStatus&facet=nameType&facet=field&facet=authorship&facet=authorshipYear&facet=extinct&facet=environment&facet=origin&issue=subspecies%20assigned&limit=100&offset=0 = NO ACTIONS IN CLB (@dhobern, whether solution should be found in TW, or Markus script should be adjusted)
Unparsable Authorship, 349. Various examples of names with problematic authorstrings https://www.checklistbank.org/catalogue/3/dataset/55434/workbench?facet=rank&facet=issue&facet=status&facet=nomStatus&facet=nameType&facet=field&facet=authorship&facet=authorshipYear&facet=extinct&facet=environment&facet=origin&issue=unparsable%20authorship&limit=1000&offset=0&reverse=false = NO ACTIONS IN CLB
Multi Word Epithet, 488. Among others, names with portion [sic]. https://www.checklistbank.org/catalogue/3/dataset/55434/workbench?facet=rank&facet=issue&facet=status&facet=nomStatus&facet=nameType&facet=field&facet=authorship&facet=authorshipYear&facet=extinct&facet=environment&facet=origin&issue=multi%20word%20epithet&limit=1000&offset=0&reverse=false = NO ACTIONS IN CLB. These are "orthographic variants". There are lot of such names in zoological GSDs. @mdoering, it would be nice, if CLB automatically remove "sic" portion and flag such names as "Orthographic Variant" in Synonym group (less preferable term "Misspelled Name").
Indetermined, 29861. Too many names to address them immediately... https://www.checklistbank.org/catalogue/3/dataset/55434/workbench?facet=rank&facet=issue&facet=status&facet=nomStatus&facet=nameType&facet=field&facet=authorship&facet=authorshipYear&facet=extinct&facet=environment&facet=origin&issue=indetermined&limit=100&offset=0 Just few examples: Lysandra i nigrum semi i nigrum Bright & Leeds, 1938 Melanargia (orlaria) Ribbe, 1909 Mnasitheus forma Evans, 1955 Terthreutis series Bai, 1993 Abantis subsp. contigua Evans, 1937 (29,844 names in this list are subspecies with missing species epithet (synonyms)) = BLOCKED
Sector(s), steps
[x] All old sectors deleted, order (subtree) Lepidoptera deleted; new sector established as order Lepidoptera from the Lepidoptera checklist (id 55434); synced 2022-05-26
[x] Nested sectors established (Replace): o Alucitoidea should be replaced with the Catalogue of World Alucitoidea = OK; synced 2022-05-26 o Pterophoroidea should be replaced with the Catalogue of World Pterophoroidea = OK; synced 2022-05-26 o Gelechiidae should be replaced with the new Gelechiidae dataset = OK; synced 2022-05-26 (need to be re-synced after assessment of new GSD) o Papilionidae, Pieridae and Lycaenidae should be replaced with the versions from GloBIS/GART = OK; synced 2022-05-26 o Gracillariidae should be replaced with the version from Global Gracillariidae = OK; synced 2022-05-27 o Tineidae should be replaced with the version from Tineidae NHM = OK; synced 2022-05-27 o Nepticuloidea should be replaced with the Nepticuloidea dataset = OK; synced 2022-05-27
[x] Appropriate taxa blocked in the Lepidoptera checklist (id 55434), 2022-05-26: o Alucitoidea o Pterophoroidea o Gelechiidae (in Gelechioidea) o Papilionidae, Pieridae and Lycaenidae (in Papilionoidea) o Gracillariidae (in Gracillarioidea) o Tineidae (in Tineoidea) o Nepticuloidea
[x] There are 75 synonyms with portion nec in the authorstring. https://www.checklistbank.org/catalogue/3/dataset/55434/workbench?facet=rank&facet=issue&facet=status&facet=nomStatus&facet=nameType&facet=field&facet=authorship&facet=authorshipYear&facet=extinct&facet=environment&facet=origin&limit=100&offset=0&q=nec @dhobern, shall we flag these names as "Misapplied Names" in the CoL?
There are also 4 synonyms with portion non (Anacampsis tremella (auctorum) non Hübner, 1803) https://www.checklistbank.org/catalogue/3/dataset/55434/workbench?facet=rank&facet=issue&facet=status&facet=nomStatus&facet=nameType&facet=field&facet=authorship&facet=authorshipYear&facet=extinct&facet=environment&facet=origin&limit=1000&offset=0&q=non&reverse=false
= FIXED 2022-05-31: flagged as Misapplied Names
TASKS 2022-05-27
The (nec) and (non) cases should all be flagged as misapplied names. Are there issues in the current LepIndex import that I should immediately attempt to fix?
Subspecies Assigned, 151. At least some of these names, CLB incorrectly parsed species names with two-word epithet as subspecies. Examples, Agrias pseudo lesoudieri Le Moult, 1926; Agrias semi rileyi Michener; Cosmopteryx sancti vincentii Walsingham, 1892 https://www.checklistbank.org/catalogue/3/dataset/55434/workbench?facet=rank&facet=issue&facet=status&facet=nomStatus&facet=nameType&facet=field&facet=authorship&facet=authorshipYear&facet=extinct&facet=environment&facet=origin&issue=subspecies%20assigned&limit=100&offset=0 = NO ACTIONS IN CLB (@dhobern, whether solution should be found in TW, or Markus script should be adjusted)
The verbatim data does place 2 words in the specificEpithet field, so we don't need to parse the names and the resulting name is just fine: https://api.checklistbank.org/dataset/55434/name/180232-0d0b0af472257a86e183be5385ee54e8
If we only had a scientificName string, the name parser provides the option of manual configurations for such cases. It is rather impossible to find a rule for those multi word epithet. I am slightly surprised to not see so many without hyphens.
But are we sure all these cases are indeed species? Agriades cuneo lunulata Tutt, 1909 or autonyms like Agriades cuneata cuneata Tutt, 1909. The issue flags potential errors and it would be good to check them all.
AAAAAAARGH! These are a complete mess. I suspect these are all supposed to be hyphenated but they are not even properly discoverable for editing inside TaxonWorks. They are in a limbo state. These could probably all be ignored for now as ridiculously obscure synonyms.
I may be able to get to these one at a time through one of the other TW interfaces. I'll try.
Here is what I have found. The situations in which these names end up in the LepIndex dataset all seem to be based on names with original hyphens, which have mysteriously disappeared when importing into LepIndex. They include the hyphen in the older NHM import into COL and in the original index cards. For example:
TaxonWorks has taken these names with hyphenated epithet and:
An example of an aberration:
An example of a species:
The zoological code says that these names should be corrected to:
This is how the names should appear in COL, with the hyphenated forms as verbatim names.
If every example was an aberration or form, I would suggest we simply automate updating them all to forms with complete epithets minus the spaces. However, some are clearly species. These are the ones that actually matter.
In COL today, these all appear, often nonsensically, as species names. If it is possible, the quickest temporary fix on the COL side may be to interpret all of these as if they still have the hyphen. This will leave the data in the same state as the current COL Checklist. Is that possible @mdoering ?
I can go through and steadily fix these names, but it will take some time. Since LepIndex does not know what rank they are (or therefore what the actual parent was), each one will need separate handling and interpretation to make them correct. It will take me time to go through and
I'm going through these names in TaxonWorks, fixing as many as I can. I've started with the families with just a few cases. The butterfly families and Lasiocampidae have many because collectors obsessed about small differences.
The biggest problem is that doing anything piecemeal in LepIndex rapidly turns into a quicksand experience. For each of these cases where aberrations and forms that include hyphens are misinterpreted as species, there are usually also several where hyphen-free names are misinterpreted in the same way. And a depressing number of LepIndex records are simple misreadings of the text on the cards that have led to the wrong character strings being recorded as names, e.g.:
Ah, well - every fix is a fix ...
More on why LepIndex and GloBIS (GART) need much more work. 19 of the species identified above with a space in the verbatim specific epithet come from the same publication by Bright & Leeds, 1938. In fact, these 19 are only a tiny part of a total of 301 aberration names described for a single species in the same publication. Every single one of these is incorrectly shown in LepIndex as a species-rank name, and these are don't come close to completing the full synonymy for the species in question. Behold how much taxonomist effort has been expended on Lysandra coridon: https://www.checklistbank.org/dataset/55434/taxon/281369
But, the most fun thing I've realised is that this over-worked species nevertheless does not appear at all in COL (even though there are >26,000 records of the species in GBIF). This is because COL imports the family Lycaenidae from GloBIS, even though the GloBIS site says that COL just takes the much more complete Pieridae and Papilionidae from that dataset. Big chunks of the Lycaenidae are therefore missing in COL. I think I need to recommend a switch to LepIndex rather than GloBIS for this family.
In COL today, these all appear, often nonsensically, as species names. If it is possible, the quickest temporary fix on the COL side may be to interpret all of these as if they still have the hyphen. This will leave the data in the same state as the current COL Checklist. Is that possible @mdoering ?
During interpretation that is not possible I am afraid. But I guess we could update the current ColDP to apply those corrections. This would cut us off from TW updates though until these fixes are also applied there. Unless we can script it all and reapply - might be doable.
Actually it is much simpler and flexible if I allow the name interpreter to insert the hyphen. It will be a new feature EPITHET_ADD_HYPHEN
that can be turned on dataset wide on demand in the settings.
OK - if that works. I've managed to process about a third of them now in TaxonWorks and was going to keep plugging away at them tonight.
I've carried on and processed them all now in TaxonWorks. I haven't worked out the species for which they would be aberrations or forms, but I have fixed all the hyphen/spaces.
@mdoering @yroskov Are there other critical aspects that should be fixed? Or should be reimport from TaxonWorks?
Are there other critical aspects that should be fixed? Or should be reimport from TaxonWorks?
@dhobern, I would suggest to do re-import. After that, I'll go ahead with a new version and do resolution of TASKS.
The (nec) and (non) cases should all be flagged as misapplied names.
= FIXED in CLB 2022-05-31: flagged as Misapplied Names
Task: all these accepted names need to be flagged as Provisionally Accepted via Workbench + RegEx Search. (Thousand(s) of names. Not able to get exact figure, because interface shows me only current and next page with 50 names per page (~2,250 finally)).
The task cannot be completed reliably due to reported problems: https://github.com/CatalogueOfLife/checklistbank/issues/1061. (1) Synonyms might be affected by incorrect decision, (2) Some names might be missed during process, (3) Long list might be not completed (now completed up to page 25, [Macaria] zozinaria 46 (final?) [Xylina] ligniplena to [Zonosoma] prunelliaria) with search pattern [[A-Za-z]
TASKS, ver 2022-04-30 / 2022-04-30, imported 2022-04-29
Resolved 2022-06-01
Synced (draft) 2022-06-10
Tests of sync results:
Families Gelechiidae (in Gelechioidea), Papilionidae, Pieridae, Lycaenidae (in Papilionoidea), Gracillariidae (in Gracillarioidea), Tineidae (in Tineoidea) are in place. Anyway, re-synced 2022-06-01.
Preview release started 2022-06-01
Import of the iteration 2022-06-02 (the same ver. 2022-04-30, as indicated in metadata)
TASKS
ACC-ACC species, same authors. 4 pairs of identical names with identical IDs(!) remain. = Odd names are blocked in CLB now.
Broken decisions, 1. = deleted.
Resolved 2022-06-02
Synced 2022-06-02
@dhobern, 2022-06-06:
I've now processed around 30 papers on Gelechiidae from Zootaxa and one from Systematic Entomology, all from the period 2018-2022. I think this should cover the most significant additions since Klaus retired. It would therefore be great if you can reprocess the Gelechiidae as well as LepIndex.
Synced 2022-06-08
Yes - please drop the empty genera. They exist for various reasons and are meaningless in COL terms, but will prove useful as we carry on cleaning up sections of the Global Lepidoptera Index, so I'd like to leave them in the underlying data source for now,
690 genera blocked (selected batch - blocked selected taxa).
Re-synced 2022-06-20.
ver. 2022-06-30 / 2022-06-30 imported 2022-07-29
TASKS
Resolved 2022-08-01:
@dhobern, TASKS report "Identical genus" contains 76 names with the scientificName value "Group" and rank "Genus": https://www.checklistbank.org/catalogue/3/dataset/55434/duplicates?category=uninomial&limit=990&minSize=2&mode=STRICT&offset=0&rank=genus&status=accepted&withDecision=false
I'll send you CSV file with the report.
Plus, 293 binomial combinations with a word “Group” as a genus.
= FIXED in the import 2022-08-02
ver. 2022-06-30 / 2022-06-30 imported 2022-08-02
TASKS
Synced 2022-08-02
Global Lepidoptera Index 2022-10-19 / 2022-10-19 (check version & date - need to be updated; metadata locked for me; = UPDATED); synced 2022-10-20
Global Lepidoptera Index 2022-10-19 / 2022-10-19; imported 2022-11-28 Donald, 2022-11-28: The GLI includes a number of new species. Synced 2022-11-28
ISSUES
TASKS
Resolved 2022-12-01
Synced 2022-12-01
Global Lepidoptera Index 0.32.1 / 2023-04-03; imported 2023-04-03
Synced 2023-04-03
Global Lepidoptera Index 0.32.2 / 2023-04-08; imported 2023-04-08
@dhobern, 2023-04-07: The GLI changes include complete updating of the tribe Micronoctuini (Noctuoidea: Erebidae: Hypenodinae), complete updates for several small families of fewer than 20 species, and modernisation of the taxonomy for Gelechioidea to reflect recent classifications.
ISSUES assessed 2023-04-10
TASKS
Broken decisions, 591: all deleted
[ ] @dhobern, There are two pairs of identical species records (at least, I do not see difference between their "species details" pages, except IDs in CLB: https://www.checklistbank.org/dataset/55434/taxon/740943 https://www.checklistbank.org/dataset/55434/taxon/740944
and
https://www.checklistbank.org/dataset/55434/taxon/440089 https://www.checklistbank.org/dataset/55434/taxon/443079
Resolved 2023-04-10:
Synced 2023-04-10
Global Lepidoptera Index 0.32.3 / 2023-05-10; imported 2023-05-10
[x] Imported: 157,797 spp (vs 157,659 spp in 0.32.2 / 2023-04-08)
[x] Metadata: OK, new - short name GLI (ver 0.32.3 / 2023-05-10)
[x] Sectors: OK, order Lepidoptera (in the Sectors report; and blocked taxa in the Assembly)
[x] Classificatiion: OK; as previous. Plus, family Sesiidae (new GSD Sesiidae https://github.com/CatalogueOfLife/testing/issues/228) blocked in GLI 2023-05-18
[x] Names with portion [GENUS NOT SPECIFIED] = 251 acc & syn blocked
TASKS
Resolved 2023-05-19:
Synced 2023-05-19
Global Lepidoptera Index 0.33.1 / 2023-05-30; imported 2023-05-31
Donald, 2023-06-01: several new papers processed for Gelechiidae and around 100 more species in GLI.
TASKS
Resolved 2023-06-02:
Synced 2023-06-02
Global Lepidoptera Index 0.33.1 / 2023-07-16; imported 2023-07-17
TASKS
Resolved 2023-07-17:
Synced 2023-07-17
Global Lepidoptera Index 0.33.1 / 2023-07-27; imported 2023-08-06
TASKS
Resolved 2023-08-11:
Synced 2023-08-11
@dhobern,
The name Acontia (Uracontia) titania Esper has no year in its authorstring. It might be Acontia titania (Esper, 1798)
There are two identical entries for Acontia (Acontia) aurelia Hacker, Legrain & Fibiger, 2008 in CLB: https://www.checklistbank.org/catalogue/3/dataset/55434/duplicates?authorshipDifferent=false&catalogueKey=3&category=binomial&limit=500&minSize=2&mode=STRICT&offset=0&status=accepted&withDecision=false
GLI 0.34.1 / 2023-08-15 re-synced 2023-08-17
Global Lepidoptera Index 0.34.2 / 2023-09-05; imported 2023-09-06
TASKS
Resolved 2023-09-06:
Synced 2023-09-06
re-synced 2023-09-12
Global Lepidoptera Index 1.1.23.269 / 2023-09-26; imported 2023-09-26
ISSUES assessed 2023-10-12:
TASKS
Resolved 2023-10-12:
Comments:
SYN-SYN species (different accepted, same authors) 549 of 551:
Synced 2023-10-12
Global Lepidoptera Index 1.1.23.316 / 2023-11-12; imported 2023-11-12
Synced 2023-11-13
LepIndex ver. "2022-04-39 / 2022-04-30" on PROD: https://www.checklistbank.org/dataset/55434/classification