search

CatalogueOfLife / testing

Editorial tests and discussion to prepare for COL releases
2 stars 0 forks source link

CoL of June 2021: plans & tests #121

Closed yroskov closed 4 months ago

yroskov commented 3 years ago

New datasets:

?? Datasets have been updated:

For next releases:

FIX UPDATE-BLOCK PROBLEM

yroskov commented 3 years ago

Mn, 2021-05-24:

yroskov commented 3 years ago

CCW:

  1. Pjotr has checked PREVIEW of 2021-05-19 and reported (eml to @yroskov of 2021-05-22):

    A search for the synomym “jaroslavi” give the proper result But a search for the synonym Tipula solstitialis gives no result.

Indeed, the system marked Tipula (Yamatotipula) solstitialis Westhoff, 1879 as bare name: https://data.catalogueoflife.org/dataset/1005/name/17778

https://github.com/CatalogueOfLife/testing/issues/120#issuecomment-847291440

  1. Dear Yuri

About

Very strange situation: the software has marked Tipula (Yamatotipula) solstitialis Westhoff, 1879 as “bare name” and excluded it from CoL. I have reported this as a bug to the programmers.

There are many more synonyms that are not found. Of course I can not check them all, but by random checking I found: Amphineurus flavoscutellatus - in Ormosia sytlifer - no Paradelphomyia abnormalis - no Ctenophora hilgendorfi - in Ctenophora nigrobasalis - in Ptilogyna longipedalis - in Tipula cinctocornis - in Tipula casta - in Tipula docilis - in

It is apparently not completely ad random, it seems that

all 96 synonyms in the genus Nephrotoma are found.
all 399 synonyms in the genus Tipula are not found

A very minor remark, while checking these synonyms I came across Tipula fasciculatus Fabricius, 1781, whereas the original description is as Tipula fasciculata https://github.com/CatalogueOfLife/testing/issues/120#issuecomment-847840086

FIXED

yroskov commented 3 years ago

World Plants:

Dear Yury, in Caryophyllaceae there is an error in genus Sabulina. Could you please exchange family data against attached file?

YR: file received, data will be updated.

UPDATED.

yroskov commented 3 years ago

From: Roskov, Yury Sent: Wednesday, May 26, 2021 16:53 To: Olaf Bánki Ower, Geoffrey Donald Markus Döring Thomas Stjernegaard Jeppesen Chantal Huijbers Cc: Dewalt, R Edward Subject: RE: COL Checklist 2021 planning

Dear Olaf and Ed,

I am happy to report that assembly of manually managed checklists has been completed by Geoff and me today (May 26th). Ed has agreed that Species Files updates will not appear in the 2021 Annual Checklist, but come in one of next monthly editions. I have kicked off re-syncs for all updated checklists. I hope, sync will be completed tonight. I am expecting, that the CoL will be released to the preview website tomorrow.

This version includes (https://github.com/CatalogueOfLife/testing/issues/121): 4 new checklists: • WoRMS Crinoidea • WoRMS Nematoda, Nemys • WoRMS Brachiopoda (it’s a replacement for Brachiopoda by Emig & al.) • WoRMS Euphausiacea (it’s a replacement for ETI Euphausiacea) 9 checklists have been updated: • Alucitoidea (5th update) • Brassicaceae • CCW • Eumycetozoa • Pterophoroidea (4th update) • ReptileDB • Systema Dipterorum • World Ferns (6th update) • World Plants (5th update) As you can see, it does not include ITIS of May (the export is not available yet) and new versions of WoRMS (their export is expected on 2021-06-01). It’s up to Olaf to decide, whether June’s CoL/AC will go ahead without these updates or will wait for them. I am expecting that ITIS & WoRMS updates might be completed by Geoff during the next week. My checks and final syncs of ITIS & WoRMS might be completed for June 15th.

Metadata for all manually assembled and updated checklists have been updated, checked and corrected in the checklistbank between January 2020 and May 2021. Metadata for all checklists imported from AC19 have been checked and corrected in the checklistbank for the release of 2020-01-10 Beta.

Yours, Yuri

yroskov commented 3 years ago

SYNCED, as reported above.

Preview release of 2021-05-27 is available at https://preview.catalogueoflife.org/

yroskov commented 3 years ago

ITIS of 2021-05-26, and 64 WoRMS of 2021-06-01 with new JAML metadata imported & synced. WoRMS re-use previously set up decisions. What'sNew list updated. Preview release kicked off.

Preview release of 2021-06-09 is available at https://preview.catalogueoflife.org/

yroskov commented 3 years ago

Dear Yuri, Things for the new sectors look good, but I did notice the placement of the following should be under Suborder Pleocyemata rather than under Suborder Not Assigned: infraorder: Astacidea Latreille, 1802 infraorder: Glypheidea Winkler, 1883

Thanks! Dave

yroskov commented 3 years ago

Hi Yuri,

Two remarks:

  1. it looks like you ignore the “organisations” we provide in the metadata YML i.e. https://preview.catalogueoflife.org/data/dataset/1044 lists only UvA, while we provide 19 institutes (from all editors)

  2. license says other, while we provide “CC-BY”

Regards, Bart

GO killed patches as suggested; all WoRMS synced 2021-06-09. YR 2021-06-10: Porifera 1044 - patch is empty. Let see on conversion result.

mdoering commented 3 years ago

Did someone tell Bart yet about the changed metadata.yaml? organisations especially is deprecated and he can place affiliations to institutions directly with the people (creator, editor, contributor)

bart-v commented 3 years ago

I don' think many people will be happy to change the metadata format each week... https://github.com/CatalogueOfLife/coldp/commit/6642c6ab32d6c46a609719d1cfc27ade65e9620e

Is there any attempt to make these things in some way stable?

mdoering commented 3 years ago

Yes. This change is actually the attempt to make it stable and release it as a fixed, first version. We have planned to do this final change for months, see our version 1 milestone here: https://github.com/CatalogueOfLife/coldp/issues?q=is%3Aissue+milestone%3A%22Public+release%22+

And consolidated the metadata model in a longer process with the ISG. See also https://github.com/CatalogueOfLife/backend/issues/989 and related issues.

@bart-v sorry to hear you are surprised by this change. I had expected that you were informed about our ColDP release plans when you first worked on the ColDP export.

Said that, we do silently still support the deprecated format. If you could migrate WoRMS to this v1 metadata format that's appreciated, but things should still be working otherwise.

bart-v commented 3 years ago

I was not aware about release plans/timings. I do get your COL-Users mailing list emails, but yeah...

Anyway, we should be able to deal with it, eventually.

mdoering commented 3 years ago

Thanks @bart-v, I thought @gdower or @yroskov had told you when working with you.

yroskov commented 3 years ago

I would expect that @bart-v, as a developer of most successful software for biodiversity data and a major data provider in CoL (64 checklists!), should be a member of IT consultation group(s) in CoL/GBIF. His opinion should be taken on board BEFORE any kind of changes in data model. I am surprised, that it was not happened in the case of metadata again.

yroskov commented 3 years ago

Preview release with two above fixes has started 2021-06-10.

CoL of 2021-06-10 is available at https://preview.catalogueoflife.org/. Awaiting deployment at the public portal.

What’s new in this edition (June 2021)? 4 new checklists:

WoRMS Crinoidea
WoRMS Nematoda, Nemys
WoRMS Brachiopoda (it’s a replacement for Brachiopoda by Emig & al.)
WoRMS Euphausiacea (it’s a replacement for ETI Euphausiacea)

71 checklists have been updated:

ITIS (9th update since AC19; new sectors: family Smarididae, superfamily Calyptostomatoidea, infraorder Stenopodidea)
Alucitoidea (5th update)
Brassicaceae
CCW
Eumycetozoa.com
Pterophoroidea (4th update)
ReptileDB
Scarabs (10th update)
Systema Dipterorum
World Ferns (6th update)
World Plants (5th update)

including automated updates for:

WoRMS Actiniaria
WoRMS Amphipoda
WoRMS Antipatharia
WoRMS Appendicularia
WoRMS Ascidiacea
WoRMS Asteroidea
WoRMS Bochusacea
WoRMS Brachypoda
WoRMS Brachyura
WoRMS Bryozoa
WoRMS Cephalochordata
WoRMS Ceriantharia
WoRMS Cestoda
WoRMS Chaetognatha
WoRMS Copepoda
WoRMS Corallimorpharia
WoRMS Ctenophora
WoRMS Cubozoa
WoRMS Cumacea
WoRMS Echinoidea
WoRMS Foraminifera
WoRMS Gastrotricha
WoRMS Gnathostomulida
WoRMS Holothuroidea
WoRMS Hydrozoa
WoRMS Isopoda
WoRMS Kinorhyncha
WoRMS Leptostraca
WoRMS Loricifera
WoRMS Merostomata
WoRMS MilliBase
WoRMS Mollusca
WoRMS Monogenea
WoRMS Mystacocarida
WoRMS Myxozoa
WoRMS Nemertea
WoRMS Octocorallia
WoRMS Oligochaeta
WoRMS Ophiuroidea
WoRMS Orthonectida
WoRMS Ostracoda
WoRMS Placozoa
WoRMS Polychaeta
WoRMS Polycystina
WoRMS Porifera
WoRMS Priapulida
WoRMS Pycnogonida
WoRMS Remipedia
WoRMS Rhombozoa
WoRMS Scleractinia
WoRMS Scyphozoa
WoRMS Staurozoa
WoRMS Tanaidacea
WoRMS Tantulocarida
WoRMS Thaliacea
WoRMS Thermosbaenacea
WoRMS Trematoda
WoRMS Turbellarians
WoRMS Xenoturbellida
WoRMS Zoantharia
yroskov commented 3 years ago

CoL of 2021-06-10 deployed to the public portal on 2021-06-14

2,026,387 acc & prov acc spp

yroskov commented 3 years ago
bart-v commented 3 years ago
  1. Underscores are within the CoL system, not in the WoRMS exports. i.e. 
    --
title: 'World List of Crinoidea'
alias: 'WoRMS Crinoidea'
  2. Same for distributions: these are being provided, but are not shown within the CoL system See https://github.com/CatalogueOfLife/testing/issues/119#issuecomment-859027140
mdoering commented 3 years ago

The WoRMS short names do not have underscores in CLB, it's just as WoRMS provided them: https://data.catalogueoflife.org/dataset/2300/about

Maybe there was a patch before or an older version got used?

yroskov commented 3 years ago

New preview release with few fixes in Systema Dipterorum kicked off 2021-06-18 at 16.30

mdoering commented 3 years ago

Good! Can we please regularly delete the private releases from last month? We had 7 releases in May & June alone. I understand you might want to compare with the previous release, but that would be June. What good are the 3 private May releases now?

yroskov commented 3 years ago

Preview 2021-06-18 https://preview.catalogueoflife.org/ looks good.

yroskov commented 3 years ago

2021-06-24: preview release started after resyncs 5 WoRMS Nemys, Crinoidea, Euphausiacea, Brachiopoda, Turbellarians (logo corrected), Systema Dipterorum (with extinct spp), ReptileDB (with acc subspp).