search

CatalogueOfLife / data

Repository for COL content
8 stars 2 forks source link

Authors of Brentids genera #677

Closed aoern closed 4 months ago

aoern commented 4 months ago

The data contents of Brentids source has been unchanged for years. In May 2024 version of CoL, all the 288 brentid genera had their authorship published. But now the authorships have disappeared again! No other updates have taken place for years. Is this done on purpose, or is it a side effect from something else?

mdoering commented 4 months ago

I habe to look to be certain, bit I suspect this is related to our accident when we merged other sources into COL, using authorships from those extended sources. I reverted that merge which should not appear in the current COL, but only show in the extended release which is not yet public. We are getting closer to some draft version to be reviewed in a few weeks, but as the Brentids source did not change authorship should not exist for now

aoern commented 4 months ago

Also the genera of ChiloBase have lost their authorships! And so do CCW, COOL, FADA Ephemeroptera, FishBase etc. etc. !?!

mdoering commented 4 months ago

This is the issue I was referring to: https://github.com/CatalogueOfLife/data/issues/668#issuecomment-2154544540

The bad ITIS merge happened on May 14th which resulted not only in additional names, but also augmented names which were missing authorships and published in references. Mainly genera. I am comparing counts of genera for each source with and without authorship between COL 24.4 (before the ITIS sync), COL 24.5 (after the ITIS sync, with additional names manually removed, but without the fix to augmented names) and the current release from June 21st:

select d.key, d.alias, 
count(*) FILTER (WHERE n.dataset_key=294826 and n.authorship is null) as april_no, count(*) FILTER (WHERE n.dataset_key=294826 and n.authorship is not null) as april_with,
count(*) FILTER (WHERE n.dataset_key=296511 and n.authorship is null) as may_no, count(*) FILTER (WHERE n.dataset_key=296511 and n.authorship is not null) as may_with,
count(*) FILTER (WHERE n.dataset_key=298708 and n.authorship is null) as june_no, count(*) FILTER (WHERE n.dataset_key=298708 and n.authorship is not null) as june_with
 from name_usage u 
 join name n on n.dataset_key=u.dataset_key and n.id=u.name_id 
 left join sector s on s.dataset_key=u.dataset_key and s.id=u.sector_key
 left join dataset d on d.key=s.subject_dataset_key
 where u.dataset_key in (294826,296511,298708) and n.rank='GENUS'
 group by 1,2 order by 2,1;

  key   |                alias                 | april_no | april_with | may_no | may_with | june_no | june_with 
--------+--------------------------------------+----------+------------+--------+----------+---------+-----------
   2317 | 3i Auchenorrhyncha                   |        4 |       5982 |      4 |     5983 |       3 |      5984
   2207 | Alucitoidea                          |        0 |         27 |      0 |       28 |       0 |        28
   1080 | BdelloideaBase                       |       38 |          0 |     10 |        1 |      11 |         0
   1161 | Brentids                             |      290 |          0 |      2 |      288 |     290 |         0
 170394 | Bryonames                            |        0 |       2717 |      0 |     2721 |       0 |      2721
   1005 | CCW                                  |        0 |        213 |      0 |      213 |       0 |       213
   1052 | COOL                                 |      361 |          0 |    347 |       14 |     361 |         0
   1146 | Carabcat                             |        0 |       4239 |      0 |     4239 |       0 |      4239
   1042 | ChiloBase                            |      400 |          0 |     27 |      373 |     400 |         0
   1113 | CilCat                               |      925 |          0 |    916 |        9 |     925 |         0
   2130 | Collembola.org                       |        0 |        697 |      0 |      697 |       0 |       697
 296427 | CunaxidBase                          |        0 |          0 |     32 |        0 |      32 |         0
   1166 | Entiminae                            |        2 |       2216 |      2 |     2216 |       2 |      2216
   1053 | Eumycetozoa.com                      |      101 |          0 |    101 |        0 |     104 |         0
   1138 | FADA Cladocera                       |      106 |          0 |    106 |        0 |     106 |         0
   1120 | FADA Ephemeroptera                   |      442 |          0 |    314 |      128 |     442 |         0
   1139 | FADA Halacaridae                     |       17 |          0 |     14 |        3 |      17 |         0
   1119 | FADA Nematomorpha                    |       18 |          0 |      0 |       18 |      18 |         0
   1047 | FADA Rotifera                        |      134 |          0 |      6 |      128 |       0 |         0
   1011 | FLOW                                 |     2385 |          0 |   1838 |      547 |    2385 |         0
   1010 | FishBase                             |     5139 |          0 |    457 |     4682 |    5139 |         0
   1201 | Fossil Ginkgoales                    |       15 |          0 |     14 |        1 |      15 |         0
  55434 | GLI                                  |       18 |      20785 |     18 |    20842 |      18 |     20844
   2362 | Gelechiidae                          |        1 |       1003 |      1 |     1004 |       1 |      1004
 185410 | Geometridae                          |        0 |       2021 |      2 |     2019 |       2 |      2019
   1046 | GloBIS (GART)                        |      116 |          0 |     87 |       29 |     116 |         0
   1049 | Global Gracillariidae                |        0 |        151 |      0 |      165 |       0 |       165
   1177 | Gymnodinium                          |        1 |          0 |      0 |        1 |       1 |         0
 265709 | Hepialidae                           |        0 |        150 |      0 |      150 |       0 |       150
   1118 | HymIS Crabronidae & Rhopalosomatidae |        8 |          0 |      6 |        2 |       8 |         0
   2141 | HymIS Pompilidae                     |      283 |          0 |    280 |        3 |     283 |         0
   1014 | ICTV                                 |     2830 |          0 |   2830 |        0 |    2830 |         0
   2007 | IRMNG                                |      132 |       6963 |    132 |     6963 |     132 |      6963
   2144 | ITIS                                 |        8 |      19529 |      8 |    19540 |       8 |     19539
   1190 | Jewel Beetles                        |      559 |          0 |    497 |       62 |     559 |         0
   1055 | LDL Neuropterida                     |     1184 |          0 |    833 |      351 |    1184 |         0
  54170 | LWS fleas                            |        0 |        314 |      0 |      314 |       0 |       314
   1144 | Lace Bugs Database                   |      284 |          0 |     16 |      268 |     284 |         0
   1096 | MOWD                                 |      404 |          0 |    403 |        1 |     403 |         0
   1148 | Microsporidia                        |      196 |          0 |    196 |        0 |     196 |         0
   1070 | Mites GSD Phytoseiidae               |       85 |          0 |     85 |        0 |      85 |         0
   1078 | Mites GSD Tenuipalpidae              |       33 |          0 |      1 |       32 |      33 |         0
   1172 | Nepticuloidea                        |       30 |          0 |     30 |        0 |      30 |         0
   1171 | PBI Plant Bug                        |     1530 |          0 |   1263 |      267 |    1530 |         0
 268676 | PaleoBioDB                           |     3761 |          0 |   3759 |        2 |    3761 |         0
   1104 | Phoronida Database                   |        3 |          0 |      1 |        2 |       3 |         0
   1054 | Psyllist                             |      274 |          0 |    267 |        7 |     274 |         0
   1199 | Pterophoroidea                       |        3 |        148 |      3 |      147 |       3 |       147
   1048 | RJB Geranium                         |        1 |          0 |      0 |        1 |       1 |         0
 298081 | RWC                                  |        0 |          0 |      0 |        0 |       0 |       137
   1008 | ReptileDB                            |     1257 |          0 |     89 |     1168 |    1258 |         0
   1061 | SF Aphid                             |      605 |       1830 |    605 |     1830 |     605 |      1830
   1169 | SF Chrysididae                       |        0 |         54 |      0 |       54 |       0 |        54
   1192 | SF Coleorrhyncha                     |        1 |         61 |      1 |       61 |       1 |        61
   1134 | SF Coreoidea                         |       29 |        813 |     29 |      813 |      29 |       813
   1158 | SF Dermaptera                        |        7 |        369 |      7 |      370 |       7 |       369
   1089 | SF Embioptera                        |        2 |        124 |      2 |      124 |       2 |       124
   1170 | SF Grylloblattodea                   |        3 |        313 |      3 |      313 |       3 |       313
   1198 | SF Isoptera                          |       10 |        387 |     10 |      387 |      10 |       387
   1173 | SF Lygaeoidea                        |      105 |       1061 |    102 |     1064 |     105 |      1061
   1062 | SF Mantodea                          |      114 |        655 |    114 |      655 |     114 |       655
   1168 | SF Mantophasmatodea                  |        0 |         20 |      0 |       20 |       0 |        20
   1021 | SF Orthoptera                        |       92 |       7398 |     93 |     7399 |      71 |      7409
   1050 | SF Phasmida                          |        2 |        750 |      2 |      750 |       3 |       755
   1065 | SF Plecoptera                        |       10 |        541 |     10 |      541 |      10 |       540
   1133 | SF Psocodea                          |        9 |       1498 |      9 |     1499 |       9 |      1499
   1167 | SF Zoraptera                         |        0 |         16 |      0 |       16 |       0 |        16
   1026 | ScaleNet                             |     1045 |          0 |   1006 |       39 |    1045 |         0
   1027 | Scarabs                              |        2 |       4552 |      2 |     4554 |       1 |      4548
   1206 | Sepidiini tribe                      |       55 |          0 |     55 |        0 |      55 |         0
  55353 | Sesiidae                             |        0 |        261 |      0 |      261 |       0 |       261
   2073 | Species Fungorum Plus                |    12744 |          1 |  12363 |      585 |   12945 |         0
   1082 | SpmWeb                               |       75 |          0 |     75 |        0 |      75 |         0
   1204 | StaphBase                            |       29 |       7294 |     27 |     7296 |      29 |      7294
   1101 | Systema Dipterorum                   |      628 |      22867 |    623 |    22872 |     573 |     22860
   1032 | TITAN                                |       32 |       5087 |     30 |     5091 |      32 |      5089
   1068 | Taxapad Ichneumonoidea               |     2592 |          0 |   2391 |      201 |    2592 |         0
   1143 | Tessaratomidae Database              |       60 |          0 |      1 |       59 |      60 |         0
   1164 | The Scorpion Files                   |        0 |        234 |      0 |      236 |       0 |       236
   1142 | The White-Files                      |      158 |          0 |    154 |        4 |     158 |         0
   1163 | The World List of Cycads             |       10 |          0 |      0 |       10 |      10 |         0
   1203 | ThripsWiki                           |        0 |        858 |      0 |      858 |       0 |       858
   1030 | TicksBase                            |       17 |          0 |      1 |       16 |      17 |         0
   1031 | Tineidae NHM                         |      306 |          0 |    305 |        1 |     306 |         0
 219318 | Tortricid.net                        |        0 |       1715 |      0 |     1715 |       0 |      1715
   1033 | Trichomycetes                        |       22 |          0 |     22 |        0 |      22 |         0
   1034 | UCD                                  |     2044 |          0 |      0 |        0 |       0 |         0
 124661 | UCD                                  |        0 |          0 |     90 |     4558 |      90 |      4532
   2256 | WCO                                  |        0 |       2499 |      0 |     2501 |       0 |      2507
   2232 | WCVP                                 |       11 |      14600 |     10 |    14601 |      11 |     14779
   2304 | WCVP-Fabaceae                        |        2 |       2026 |      2 |     2026 |       2 |      2026
   2004 | WFO Plant List 2024-06               |        0 |        170 |      0 |      170 |       0 |       170
 125101 | WOL                                  |        2 |        830 |      2 |      830 |       2 |       830
   1029 | WSC                                  |     4358 |          0 |    523 |     3835 |    4358 |         0
   1039 | WTaxa                                |      119 |       5462 |    117 |     5464 |     119 |      5461
   1162 | WWW                                  |        8 |          0 |      3 |        5 |       8 |         0
   1176 | WoRMS Actiniaria                     |      139 |        345 |    126 |      358 |     139 |       345
   1202 | WoRMS Amphipoda                      |        7 |       2150 |      7 |     2153 |       7 |      2153
   1194 | WoRMS Antipatharia                   |        1 |         53 |      1 |       53 |       1 |        53
   1178 | WoRMS Appendicularia                 |        1 |         18 |      1 |       18 |       1 |        18
   1186 | WoRMS Ascidiacea                     |        7 |        350 |      7 |      350 |       7 |       350
   1095 | WoRMS Asteroidea                     |        1 |        734 |      1 |      735 |       2 |       735
   1086 | WoRMS Bochusacea                     |        0 |          3 |      0 |        3 |       0 |         3
   2299 | WoRMS Brachiopoda                    |        0 |        771 |      0 |      788 |       0 |       949
   1087 | WoRMS Brachypoda                     |        0 |          5 |      0 |        5 |       0 |         5
   1108 | WoRMS Brachyura                      |        0 |       2711 |      0 |     2711 |       0 |      2713
   1081 | WoRMS Bryozoa                        |       12 |       2328 |     12 |     2328 |      12 |      2326
   1154 | WoRMS Cephalochordata                |        2 |          7 |      2 |        7 |       2 |         7
   1179 | WoRMS Ceriantharia                   |        1 |         42 |      1 |       42 |       1 |        42
   1127 | WoRMS Cestoda                        |        3 |        685 |      3 |      686 |       3 |       686
   1132 | WoRMS Chaetognatha                   |        1 |         25 |      0 |       26 |       1 |        25
   1191 | WoRMS Copepoda                       |       16 |       3004 |     16 |     3006 |      16 |      3009
   1195 | WoRMS Corallimorpharia               |        2 |         17 |      1 |       18 |       2 |        17
   2300 | WoRMS Crinoidea                      |        1 |        266 |      1 |      266 |       1 |       266
   1180 | WoRMS Ctenophora                     |        1 |        102 |      0 |      103 |       0 |       103
   1181 | WoRMS Cubozoa                        |        0 |         20 |      0 |       20 |       0 |        20
   1058 | WoRMS Cumacea                        |        6 |        161 |      5 |      162 |       6 |       161
   1106 | WoRMS Echinoidea                     |      128 |       1515 |    125 |     1518 |     125 |      1518
   2301 | WoRMS Euphausiacea                   |        0 |         19 |      0 |       19 |       0 |        19
   1157 | WoRMS Foraminifera                   |       46 |       5199 |     46 |     5202 |      46 |      5205
   1122 | WoRMS Gastrotricha                   |        6 |         78 |      3 |       81 |       6 |        78
   1125 | WoRMS Gnathostomulida                |       12 |         16 |      1 |       27 |      12 |        16
   1107 | WoRMS Holothuroidea                  |        6 |        453 |      4 |      455 |       6 |       453
   1112 | WoRMS Hydrozoa                       |       10 |       1097 |      9 |     1098 |      10 |      1098
   1094 | WoRMS Isopoda                        |        2 |       2134 |      2 |     2136 |       3 |      2139
   1153 | WoRMS Kinorhyncha                    |        1 |         49 |      0 |       50 |       1 |        49
   1105 | WoRMS Leptostraca                    |        0 |         13 |      0 |       13 |       0 |        13
   1182 | WoRMS Loricifera                     |        0 |         16 |      0 |       16 |       0 |        16
   1152 | WoRMS Merostomata                    |        0 |          5 |      0 |        5 |       0 |         5
   1200 | WoRMS MilliBase                      |       63 |       3421 |     61 |     3422 |      61 |      3421
   1130 | WoRMS Mollusca                       |      530 |      20715 |    515 |    20776 |     525 |     20845
   1126 | WoRMS Monogenea                      |        6 |        955 |      6 |      956 |       6 |       956
   1088 | WoRMS Mystacocarida                  |        0 |          2 |      0 |        2 |       0 |         2
   1129 | WoRMS Myxozoa                        |        0 |         92 |      0 |       98 |       0 |        97
   1085 | WoRMS Nemertea                       |       36 |        402 |     37 |      401 |      36 |       403
   2302 | WoRMS Nemys                          |      365 |       2767 |    363 |     2776 |     363 |      2776
   1131 | WoRMS Octocorallia                   |       11 |        690 |     10 |      693 |      10 |       693
   1099 | WoRMS Oligochaeta                    |        2 |        777 |      2 |      777 |       2 |       777
   1059 | WoRMS Ophiuroidea                    |       10 |        444 |     10 |      444 |      10 |       444
   1149 | WoRMS Orthonectida                   |        4 |          2 |      0 |        6 |       4 |         2
   1175 | WoRMS Ostracoda                      |       61 |       3793 |     60 |     3795 |      61 |      3794
   1123 | WoRMS Placozoa                       |        0 |          4 |      0 |        4 |       0 |         4
   1090 | WoRMS Polychaeta                     |       34 |       2504 |     32 |     2507 |      34 |      2507
   1109 | WoRMS Polycystina                    |      221 |         28 |    221 |       28 |     221 |        28
   1044 | WoRMS Porifera                       |       51 |       1680 |     51 |     1680 |      52 |      1679
   1124 | WoRMS Priapulida                     |        0 |          9 |      0 |        9 |       0 |         9
   1183 | WoRMS Pycnogonida                    |        3 |        136 |      0 |      139 |       3 |       136
   1091 | WoRMS Remipedia                      |        0 |         12 |      0 |       12 |       0 |        12
   1150 | WoRMS Rhombozoa                      |        2 |          7 |      0 |        9 |       2 |         7
   1196 | WoRMS Scleractinia                   |       30 |       1734 |     30 |     1735 |      30 |      1735
   1188 | WoRMS Scyphozoa                      |        1 |         94 |      1 |       94 |       1 |        94
   1184 | WoRMS Staurozoa                      |        3 |         25 |      3 |       25 |       3 |        25
   1103 | WoRMS Strepsiptera                   |        0 |        114 |      0 |      114 |       0 |       114
   1110 | WoRMS Tanaidacea                     |        5 |        391 |      5 |      393 |       5 |       394
   1092 | WoRMS Tantulocarida                  |        0 |         24 |      0 |       24 |       0 |        24
   1185 | WoRMS Thaliacea                      |        4 |         37 |      4 |       37 |       4 |        37
   1093 | WoRMS Thermosbaenacea                |        1 |          7 |      1 |        7 |       1 |         7
   1128 | WoRMS Trematoda                      |        3 |       2587 |      3 |     2614 |       3 |      2623
   1193 | WoRMS Turbellarians                  |       23 |       1356 |     20 |     1358 |      24 |      1347
   1100 | WoRMS Xenoturbellida                 |        0 |          1 |      0 |        1 |       0 |         1
   1197 | WoRMS Zoantharia                     |        0 |         38 |      0 |       38 |       0 |        38
   1140 | World Ferns                          |        0 |        396 |      0 |      396 |       0 |       396
   1141 | World Plants                         |        1 |       8703 |      0 |     8704 |       2 |      8692
   1037 | ZOBODAT Vespoidea                    |      320 |          0 |    309 |       11 |     320 |         0
        |                                      |     1466 |       1075 |    684 |     5724 |    1466 |      1063
(165 rows)

There were a few source changes in between too

mdoering commented 4 months ago

large changes were basically all old sources that never supplied genus authorships. Potentially because they are still stuck with the old ACEF format which did not support that:

  key   |                alias                 | april_no | april_with | may_no | may_with | june_no | june_with 
   1161 | Brentids                             |      290 |          0 |      2 |      288 |     290 |         0
   1052 | COOL                                 |      361 |          0 |    347 |       14 |     361 |         0
   1042 | ChiloBase                            |      400 |          0 |     27 |      373 |     400 |         0
   1120 | FADA Ephemeroptera                   |      442 |          0 |    314 |      128 |     442 |         0
   1011 | FLOW                                 |     2385 |          0 |   1838 |      547 |    2385 |         0
   1010 | FishBase                             |     5139 |          0 |    457 |     4682 |    5139 |         0
   1190 | Jewel Beetles                        |      559 |          0 |    497 |       62 |     559 |         0
   1055 | LDL Neuropterida                     |     1184 |          0 |    833 |      351 |    1184 |         0

   more...

In any case I cannot detect a source that has changed from the April edition. May was the corrupted one and it looks like we reverted back just fine.

mdoering commented 4 months ago

@yroskov @aoern can we close the issue?

yroskov commented 4 months ago

Missing authorstrings with taxa above species in ACEF checklists, as expected and as explained by Markus

aoern commented 4 months ago

It is a pity that we cannot publish thousands of good authorship strings!

mdoering commented 4 months ago

they will be published every month soon in the extended release