TITAN (id 1032): test report

yroskov commented 1 year ago

TITAN DEV2 https://www.dev.checklistbank.org/dataset/55378/classification

Version crawled from the website http://titan.gbif.fr/.

Imported 2023-06-29:

Imported: 36,175 spp, 5830 genera (vs 33,954 spp in Dec 2004) ("La base Titan contient 38,714 espèces, 19,184 références bibliographiques et 52,992 types")

Classification:

Tribe nec Cerambycoidea (subfamily Phytophaga)

yroskov commented 1 year ago

TITAN DEV3 https://www.dev.checklistbank.org/dataset/55400/classification

Imported 2023-06-30:

Imported: 36,175 (vs 36,175) spp, 5108 (vs 5830) genera

yroskov commented 12 months ago

(TITAN DEV3)

[x] Classification: family Cerambycidae + subgenus Exocentrus (Exocentrancylistes) Breuning, 1965 with one species Exocentrus (Exocentrancylistes) villiersi in the root of the Tree - subfamilies (17) - tribes (267) - subtribes (25) - genera (5107) - species - subspecies (2482), there are no type subspecies in CLB checklist version

...there are no type subspecies in CLB checklist version due to presence of "=" on subspecies page (such pages were excluded, because they create incorrect species placements in genera):

In TITAN (http://titan.gbif.fr/sel_synonyme.php?numero=12081&id_nom_synonyme=305904&nom_synonyme=Acalolepta%20(Dihammus)%20rusticator%20rusticator):

yroskov commented 12 months ago

[ ] Misplaced taxa: number of subtribes misinterpreted as tribes in CLB Tree. (Tribes always have ending -ini)

SUGGESTION: ignore portions in brackets during conversion (i.e. no subtribes)

(Example: tribe Cerambycina & tribe: Cerambycinae)

In TITAN, the list of tribes:

In CLB:

Other cases:

In TITAN:

in CLB:

In TITAN:

in CLB:

In TITAN:

in CLB:

In TITAN:

yroskov commented 12 months ago

[ ] For YR: taxa should be blocked from CoL subfamily Phytophaga with tribe nec Cerambycoidea and 9 genera subfamily Composites with 3 genera & 3 spp

tribe nomina nuda with 7 genera = species not present in CLB tribe Fossiles with 33 genres - it present in TITAN = not in CLB. In theory, these 33 genera (http://titan.gbif.fr/sel_tribu.php) should be attached directly to the family Cerambycidae, but we may get split genera. Thus, species should be attached directly to the family Cerambycidae (i.e. without parent genus) with "Accepted" status.

[x] ~~For YR:~~ empty tribes should be blocked from CoL. Geoff reported empty tribes Onocephalini & Mallodonini (error page in TITAN; empty tribes in CLB tree). Their children genera moved to other tribes. Empty tribes should be blocked in Assembly. = FIXED in DEV4, 2nd iteration

yroskov commented 12 months ago

[x] Synonyms & combinations: subsequent combinations have no authorstrings in synonymic list. (This simplification at this stage agreed).

Example: Aegomorphus pereirai (Prosen & Lane, 1955)

In CLB: https://www.dev.checklistbank.org/dataset/55400/taxon/121

In TITAN: http://titan.gbif.fr/sel_genann1.php?numero=121

yroskov commented 12 months ago

[x] Distribution: no distribution data at this stage. (Distribution is present in TITAN per reference)

yroskov commented 12 months ago

TITAN DEV4 https://www.dev.checklistbank.org/dataset/56387/classification

Imported 2023-07-11, 1:39 PM

[x] Imported: 36,175 (vs 36,175) spp

[ ] Classification: subgenus Exocentrus (Exocentrancylistes) in the root of the Tree (out of family) = Geoff: most of these subgenera are placed in Exocentrini, except for that one subgenus Exocentrus (Exocentrancylistes) which is in Acanthocinini. = YR: should be blocked from CoL

TITAN website listed in "browse tribes" 249 true tribes, plus 4 other entries Composites (3 genera with 3 species) = should be blocked in CoL Fossiles = will be resolved in ~~DEV5~~ DEV4 2nd iteration Incertae sedis (Cerambycinae) Incertae sedis (Lamiinae) Incertae sedis (Prioninae) = will be resolved in ~~DEV5~~ DEV4 2nd iteration nec Cerambycoidea = should be blocked in CoL

yroskov commented 12 months ago

TITAN DEV4 https://www.dev.checklistbank.org/dataset/56387/classification - new iteration

Imported 2023-07-11, 5:37 PM

[x] Imported: 36,216 (vs 36,175) spp

[x] Flag "extinct" is missing = FIXED in 2023-07-12
[x] Extinct species are outside the family Cerambycidae = FIXED in 2023-07-12
[ ] "Subtribe Lamiinae" is outside tribe Lamiini. Children genera of "subtribe Lamiinae" should go to subfamily Lamiinae
[ ] "Subtribe Cerambycinae" is outside tribe Cerambycini. Genera from "subtribe Cerambycinae" should go to subfamily Cerambycinae
[ ] "Subtribe Prioninae" is outside tribe Prionini. Genera from "subtribe Prioninae" should go to subfamily Prioninae

yroskov commented 12 months ago

TITAN DEV4 https://www.dev.checklistbank.org/dataset/56387/classification - next iteration

Imported 2023-07-12, 4:04 PM

[x] Imported: 36,216 (vs 36,216) spp
[x] Three Incertae sedis tribes are not resolved yet: tribes Incertae sedis and their children subtribes "Cerambycinae", "Lamiinae" & "Prioninae" need to be killed as intermediate ranks. Their genera should be attached directly to their parent subfamilies. = see fix in DEV2, not fully fixed; in DEV4

yroskov commented 12 months ago

TITAN DEV4 https://www.dev.checklistbank.org/dataset/56387/classification - new iteration

Imported 2023-07-12, 8:29 PM

[x] Imported: 36,216 (vs 36,216) spp
[x] Three Incertae sedis tribes: genera moved to a next parent (Fix!); 3 empty Incertae sedis tribes should be blocked in Assembly (YR!)

NEXT STEPS IN PRODUCTION:

[ ] Composites (3 genera with 3 species) = should be blocked in Assembly
[x] 3 empty Incertae sedis tribes should be blocked in Assembly
[ ] nec Cerambycoidea = should be blocked in Assembly

yroskov commented 11 months ago

TITAN DEV4 https://www.dev.checklistbank.org/dataset/56387/classification - another iteration

Imported 2023-07-12, 11:31 PM

[x] Imported: 36,216 (vs 36,216) spp
[x] Three Incertae sedis tribes = FIXED (no Incertae sedis, children genera attached directly to subfamilies
[x] Empty tribes Onocephalini & Mallodonini (error page in TITAN; empty tribes in CLB tree) = reported by Geoff above

NEXT STEPS IN PRODUCTION:

[ ] Composites (3 genera with 3 species) = should be blocked in Assembly
[ ] nec Cerambycoidea = should be blocked in Assembly

yroskov commented 11 months ago

Copy of ACCESS database received 2023-07-25.

yroskov commented 11 months ago

TITAN of Jun 2023 / 2023-06-26 imported to the production CLB 2023-07-13

[x] Imported: 36,216 spp
[x] Metadata: updated New abstract (YR): The database represents more than 99% of valid described species and subspecies of world’s Cerambycidae. It contains over 38K species, 19K bibliographic references and 53K types. When it's possible, the authors included information on geographic distribution, host and visited plants, parasites, predators, mimics, and common names in different languages. All suggestions for enrichment and improvement of the content are welcome. This version of data has been crawled in CoLDP format from the original TITAN website by Geoff Ower, CoL Database Manager.
[x] Classification: OK. Subgenus Exocentrus (Exocentrancylistes) is in the Tree root outside family Cerambycidae. Empty tribes Onocephalini & Mallodonini (error page in TITAN; empty tribes in CLB tree) = reported above
[x] Sector: family Cerambycidae re-established 2023-08-08
[x] subfamily Composites (3 genera with 3 species) = blocked in Assembly
[x] tribe nec Cerambycoidea = blocked in Assembly
[ ] Set of synonyms without authorstrings: can we address it having copy of Access database?

ISSUES assessed 2023-08-08

Multi Word Monomial, 520: 520 accepted subgenera recorded in combination with genus name. https://www.checklistbank.org/catalogue/3/dataset/1032/workbench?facet=rank&facet=issue&facet=status&facet=nomStatus&facet=nameType&facet=field&facet=authorship&facet=authorshipYear&facet=extinct&facet=environment&facet=origin&issue=wrong%20monomial%20case&limit=800&offset=0 Examples: Abatocera (Sternobatocera) Breuning, 1943; Acalolepta (Pilohammus) Vitali, 2019; Bacchisa (Cyanastus) Pascoe, 1867 @gdower, it's not clear to me whether CLB able to handle subgenera in such form or not.
names (all synonyms) with words "subgen.", "morpha", "subsp.", "sp." (3,246 names with "sp") blocked. These names should be blocked BEFORE resolving Tasks

TASKS

Broken decisions, 4944: deleted all

Resolved 2023-08-08:

Sync started 2023-08-08 Sync failed:

yroskov commented 11 months ago

@olafbanki, sync of new TITAN data failed due to "OTHER" license. Indeed, metadata says "License: Other" I didn't find TITAN in the log of license negotiations https://github.com/CatalogueOfLife/testing/issues/30 Could you please confirm the license for TITAN? (Data are ready for August edition, but sync is blocked by the CLB due to license mismatch).

yroskov commented 10 months ago

[x] I changed TITAN license to CC-BY 2023-08-17

@olafbanki, 2023-08-15: With the removal, I assume also the metadata changed; as I explicitly changed the licensing of TITAN for the Annual Checklist 2023. TITAN should be CC-BY. This agreement was reached with Thierry Bourgoin, after we and him did not manage to get in contact with Gerard (contact not established since 2021).

Synced 2023-08-17

yroskov commented 10 months ago

TITAN of Jun 2023 / 2023-06-26 imported to the DEV 2023-08-23 8:48 PM as titan dev2 after adjustments in the crawler script

to fix subgenera presentation
to bring back authorstrings in combinations in synonymy
to bring distribution data

Tested 2023-08-24 at https://www.dev.checklistbank.org/dataset/55378/classification

[x] Imported: 36210 spp (vs 36,216 spp in previous import)
[ ] Classification: subgenus Exocentrancylistes Breuning, 1965 (with one species) is outside the genus and outside family Cerambycidae

[x] to fix subgenera presentation = FIXED Example: Exocentrus

It was a month ago (view in the portal):

It is now (view in the DEV CLB):

[ ] to bring back authorstrings in combinations in synonymy = @gdower, I'm not sure if the issue is fixed in this iteration: https://www.dev.checklistbank.org/dataset/55378/taxon/1287 https://www.dev.checklistbank.org/dataset/55378/taxon/22127 https://www.dev.checklistbank.org/dataset/55378/taxon/28277 https://www.dev.checklistbank.org/dataset/55378/taxon/13725
[x] to bring distribution data = FIXED (agreed compromise: text lines as they appear on the source website). There is also normalized & standardized distribution on the website (see below).

yroskov commented 10 months ago

[ ] bring back authorstrings in combinations in synonymy Species for tests (with multiple protonyms): https://www.dev.checklistbank.org/dataset/55378/taxon/1287 https://www.dev.checklistbank.org/dataset/55378/taxon/4133 https://www.dev.checklistbank.org/dataset/55378/taxon/2299 https://www.dev.checklistbank.org/dataset/55378/taxon/4997 https://www.dev.checklistbank.org/dataset/55378/taxon/548 https://www.dev.checklistbank.org/dataset/55378/taxon/549
Attention:

https://www.dev.checklistbank.org/dataset/55378/taxon/6807 ≡ Anoploderma (Pathocerus) humboldti Lameere, 1912
= Anoploderma humboldti (Lameere, 1912) = should be no brackets

https://www.dev.checklistbank.org/dataset/55378/taxon/17280 = Logisticus oberthueri (Fairmaire, 1889) = should be no brackets (ortho variant) = Logisticus oberthurii Fairmaire, 1889

yroskov commented 10 months ago

[ ] add distribution data

Normalized & standardized distribution (incl. ISO codes) per species is available on the website:

http://titan.gbif.fr/sel_pays1.php?numpays=4133:

yroskov commented 10 months ago

TITAN dev imported for tests 2023-08-29; 2023-08-30 in CLB https://www.checklistbank.org/dataset/264420/imports

[x] Imported: 36210 spp (vs 36210 spp)

yroskov commented 10 months ago

Possible further improvements:

[x] Authorstrings for the tribes: present at the bottom of tribe pages, in bold. For example: http://titan.gbif.fr/sel_tribu.php?nomtribu=Caraphiini. Unfortunately, many names with bold authors do not have exact match with tribe name (examples Acanthocinini, Acanthoderini, Curiini, etc.). Imossible to fill tribe authorstrings.
[x] Authorstrings for the subfamily: I didn't find it neither on the website nor in Access

yroskov commented 10 months ago

TITAN dev imported for tests 2023-08-30 in CLB https://www.checklistbank.org/dataset/264420/imports (all data have been re-crawled; distribution data have been added)

Imported to CLB-prod 2023-08-31 as TITAN of Aug 2023 / 2023-08-29 (id 1032).

[x] Imported: 36371 spp (vs 36210 spp)
[x] Metadata: OK
[x] Classification: OK, see above
[x] Sector: OK

[x] Species page: [x] authorstrings in synonymy; [x] distribution Sample spp: Acanthocinus aedilis (Linné, 1758) https://www.checklistbank.org/dataset/264420/taxon/10462 Alexera barii (Jekel, 1861) https://www.checklistbank.org/dataset/264420/taxon/455
[x] block names with portions "subgen.", sp., subspecies, rasse, morpha etc, see also Multi Word Epithet & Parsed Name Differs

ISSUES assessed 2023-08-31

TASKS

Broken decisions, 40: deleted all

@gdower, it's a new "feature" (perhaps, something wrong also with IDs/parent-child relationships - I cannot block synonyms in CLB):

[ ] There are many (21,166) synonyms identical to its parent species: ACC-SYN species (same accepted, same authors) 1 of 21166 = I am unable to block them in CLB:

The same problem with subspecies: ACC-SYN infraspecies (same accepted, same authors) 328 of 1310

The same problem with duplicated synonyms vs accepted names in the report SYN-SYN species (same accepted, same authors) 16 of 48 = FIXED through blocking in CLB

Resolved 2023-08-31:

Sync 2023-08-31, 13.45-14.00 (Champaign) failed

yroskov commented 10 months ago

https://github.com/CatalogueOfLife/backend/issues/1246

yroskov commented 10 months ago

For attention of @mdoering: Sync 2023-09-01, 7.58 (Champaign) again failed

yroskov commented 10 months ago

TITAN of Aug 2023 / 2023-08-29 (id 1032) re-imported 2023-09-01.

TASKS

Broken decisions, 1940: deleted all

Resolved 2023-09-01:

Sync 2023-09-01 = failed

mdoering commented 10 months ago

The last syncs claim to have been canceled, not failed: https://www.checklistbank.org/catalogue/3/sector/sync?sectorKey=1996

The logs contain this suspicous entry:

ATTACH taxon tree ACCEPTED FAMILY Cerambycidae Latreille, 1802 [Cerambycidae] to SUPERFAMILY Chrysomeloidea [b1bcfd7d-b704-40b4-bf59-0bd6484330d8]. Blocking 6009 nodes

Blocking 6009 nodes? There are ~9500 decision in place, mostly blocks. Maybe thats real then. I will look into this on Monday, no immediate idea why that is

yroskov commented 2 months ago

TITAN of 2023-12-22 / 2023-12-22; imported 2024-04-23

[x] Imported: 36371 spp (vs 36210 spp)
[x] Metadata: version changed 2023-12-22 / 2023-12-22 -> Dec 2023 / 2024-04-23
[x] Classification: OK; subgenus Exocentrancylistes is outside family Cerambycidae
[x] Sector: OK
[x] Species page: [x] authorstrings in synonymy; [x] distribution (checked Derolus arciferus; Acanthocinus aedilis; Alexera barii)
[x] block names with portions "subgen.", sp. (+38), subspecies, rasse (+1), morpha etc, see also Multi Word Epithet & Parsed Name Differs

Metrics

ISSUES assessed 2024-05-01

TASKS

Broken decisions, 46 = deleted
Outdated decisions, 10 = deleted

Resolved 2024-05-01:

Synced 2024-05-01

CatalogueOfLife / testing

TITAN (id 1032): test report #231