Publish the UiO Paleontology collection on our IPT

[dagendresen]

Eirik has a fresh copy of the NORPAL collection database (in dBASE, FoxPro format)

A strategy could be to (1) publish the PAL base (as is) in Zenodo (following our described routine), and (2) extract what we succeeded to map to Darwin Core and publish on the IPT.

Can we meet on Friday at 10 (before the all-staff meeting at 11:15; and Peters disputation at 13:15)? To start preparing the dataset for publication?

[rukayaj]

Just to keep this issue up to date, @MichalTorma volunteered to take a look at this (I think you did right!?).

• There are two files, one with a list of occurrences which are type specimens, one with a list of occurrences which are ordinary specimens - there are different fields in both files, but no overlap of records (is this correct?)
• We will first make an attempt to separate out the names + dates which are currently contained in one field.
  • There is a lot variation in this field so we are going to have to make some decisions about the type of patterns we will work with and those we won't, see also https://github.com/gbif-norway/data-prep-scripts/blob/fossil-db/scripts/fossil-db/tests.py
• All the other fields we decided to pretty much publish as-is, except I suppose we could do some very basic stuff like remove the '0's in the lat + long fields... anything else?
• We decided we would try and do some roundtripping (getting our corrected data back into the source foxpro database). So this would mean just the dates + names, I suppose?
• We decided to drop ALL the records which have duplicated PMO_NR and sub-no
• We decided to publish our occurrences with UUIDs, and map those UUIDs for future updates of the database using PMO_NR and sub-no (these should now be unique, as we dropped all duplicates)
• We will do all data cleaning (inc duplicate dropping) programmatically, in a way which means we can easily repeat it when we do the next import of this database

Did we decide anything about adding materialSampleID?

Please add in here if there's anything I've forgotten.

[dagendresen]

These are all MaterialSamples!! -- so I suggest to not bothering with creating any occurrenceID UUIDs - but maybe use DwC-triplets here? Creating FALSE occurrenceID identifiers will only cause us severe headaches later... ;-)

But yes to create and add urn:uuid:UUIDs as materialSampleID

[dagendresen]

Another question might be if bothering to create the "parent MaterialSample" from each PMO_NR without the sub-no -- for the actual rock that (I assume) includes 1 to many fossils (as enumerated by the sub-no).

I suggest to NOT create these "parent MaterialSample"s - until possibly next round - and in coordination (!) with Hans Arne.

[rukayaj]

Some more notes:

Main spreadsheet:

• check the kingdom values - some stuff needs to get organised here, some we can just ignore (because of gbif's auto mapping)
• maybe put some of the localities into one of the geolocating tools, when there is no decimal degrees info
• geologicalContext -> occurrenceRemarks ?
• remarks -> also needs to go somewhere else...

Types spreadsheet:

• same kingdom problem
• swap fylke - > stateprovince, county is kommune (it is correct in the other file)
• geologicalContext is also problematic... maybe lithostratigraphicTermsProperty ?
• typeStatus - Eirik has asked about this as they are rather cryptic, 'H' is possibly holotype? hopefully we will get back sensible values we can map
• references -> materialCitation probably... these are the info about where the type specimen is actually published

[rukayaj]

For the Type field: The TYPE field is intended to specify real types, ie parts of the type series: H = holotype P = para (lecto) type L = lectotype N = neotype S = syntype

"T" should not be found here. At least I could not find any.

"R" must be an error. Thanks for making me aware of that. Have a suspicion of what may have happened. Must check this more closely and will probably delete these Rs.

[rukayaj]

Just to update this, we have something now that is working quite well for the date/name extraction, and the rest of the data cleaning is minor and should be fairly easy.

[rukayaj]

https://ipt.gbif.no/manage/resource.do?r=o_fossils

[rukayaj]

I would say that these two files (type specimens and ordinary specimens) should get published as a single dataset seeing as they will have pretty much the same metadata - any thoughts?

[dagendresen]

Sounds reasonable -- however, can you make sense of the non-type table? Should these records be published at all? Are the records persistently identifiable and persistently linked to the physical specimens/samples?

[rukayaj]

Sounds reasonable -- however, can you make sense of the non-type table? Should these records be published at all? Are the records persistently identifiable and persistently linked to the physical specimens/samples?

Yes, Eirik says the SUBNO and PMO_NR will remain consistent, so we will be able to link them up again. We dropped all of the ambiguous ones, so we're only publishing the ones that meet certain criteria.

It's all ready to be published now, just waiting for him to fill in the metadata.

[rukayaj]

Followed up with him on 11 May

[rukayaj]

Published :) https://www.gbif.org/dataset/b2522b78-18ec-4ba6-ba16-9c9e215ce9e6

[rukayaj]

See https://github.com/gbif/portal-feedback/issues/4291, I forgot to add a grscicoll collection code. And Morten found some more, so I'll do a bit more tidying up here when I'm back.

[rukayaj]

According to Eirik it should be PMO

[dagendresen]

Apropos - "O" is only for the herbarium, and should probably not be in the dataset title. PPS: All the locations in the south seem to be wrong --> Svalbard and Canada (not Africa and India)

[dagendresen]

Removed the "O" from the dataset title and added a hard-mapped institutionID mapping to the ROR identifier, https://ror.org/01xtthb56

[rukayaj]

All the locations in the south seem to be wrong --> Svalbard and Canada (not Africa and India)

Yes, we sent Hans Arne some feedback asking about this and some other issues, but didn't get a reply.

[rukayaj]

We are very happy to tell you the palaeontological collection has now been published to GBIF - https://doi.org/10.15468/wmmvk3 . This DOI can be used to track publications which use data from this dataset, and I notice there is actually already somebody who has downloaded some data points from us (https://www.gbif.org/dataset/b2522b78-18ec-4ba6-ba16-9c9e215ce9e6/activity).

We have published the type specimens (23809) and non-type specimens (178247) together. In total 202 056 occurrences.

There are a few issues we noticed:

There are a lot of latitude and longitude coordinates of '-9' , which I guess is wrong? Should we assume that these coordinates are invalid and remove them before we publish? There is a tabular list of the affected records here: https://www.gbif.org/occurrence/search?dataset_key=b2522b78-18ec-4ba6-ba16-9c9e215ce9e6&issue=COUNTRY_COORDINATE_MISMATCH There are some records where GBIF did not recognise the taxon - https://www.gbif.org/occurrence/search?offset=0&dataset_key=b2522b78-18ec-4ba6-ba16-9c9e215ce9e6&issue=TAXON_MATCH_NONE . Perhaps this is ok in most cases, but it might be helpful to use this information to make some corrections to the source database? Of the non-type specimens we couldn't publish 3901 records, and of the types we couldn't publish 4017 records. This is because it was not possible to make a unique combination of the PMO_NR and SUBNO columns. We could drop in and talk to you about this if you're available tomorrow afternoon?

Please have a look and see if there are any other issues we should correct (Eirik, can you take a look too please?). We can publish as many corrections as we choose, and we should publish regular updates - so should that be one every six months as specified in the metadata?

[MichalTorma]

Extinct taxa in an extant world https://doi.org/10.3897/biss.6.94417 - it might be relevant to pass this talk on when it becomes public

[rukayaj]

From holly little's talk about wikidata

[dagendresen]

I have been imagining we could explore more if possible to record collecting sites in DEIMS...?

(DEIMS is focused on a catalog of LTER monitoring stations --- but the DEIMS developer (Christoph Wohner) suggested (during the BioDT WS in Potsdam last month) that DEIMS could also handle collecting sites (sites which are not yet revisited -- on the rationale that they could be..).