how does eml-gbif relate to eml

[pvgenuchten]

We're in the proces of creating a metadata-schema-plugin for EML. We're a bit puzzled about the various versions numbers used. Can you share a bit of history or links to documentation on how the current standards evolved

In this repository, there is a schema https://github.com/gbif/eml-profile/blob/master/eml-gbif-profile.xsd which is quite different then the schema at https://github.com/gbif/rs.gbif.org/tree/master/schema/eml-2.1.1

What is the relation of the two schema's.

For the 2.2.0 EML schema, do you envision a similar implementation of the gbif profile?

[timrobertson100]

Thanks @pvgenuchten - those questions really jog the memory as this was all done in around 2009 I think.

The people who did this aren't here to ask anymore but I recall that EML was itself releasing 2.1.x at that time, and there were some issues that led us to hosting a version of the EML xsd. I could ping Matt Jones and see if he can remember if it helps.

The GBIF EML profile was developed as a small subset of EML and then extended following the recommendations at the time in the additionalMetadata element. There is some text around it here. To my knowledge, when it was first released a GBIF metadata document would always validate against the EML schema too.

For the 2.2.0 EML schema, do you envision a similar implementation of the gbif profile?

We have no plans to, but I would assume at some point we will be bumping to a newer EML. I suspect it has changed significantly so we'd follow whatever recommendations there are at the time.

Are you just exploring, or are you hitting real issues please, in which case we can spend time investigating too?

[pvgenuchten]

Hi @timrobertson100, thanx for the quick reply. We try to update the geonetwork schema-plugin to 2.2.0, but we have some challenges with the 2.2.0 xsd. We noticed that https://github.com/gbif/eml-profile/blob/master/eml-gbif-profile.xsd works out of the box in GeoNetwork.

[timrobertson100]

Mmm... I am sorry I am not more helpful. There was a genuine reason we ended up hosting XSDs but I am afraid that escapes me. @mdoering do you recall anything around this, please?

[mdoering]

The GBIF EML profile is a subset of the full EML. So our schema cherry picks the stuff we thought is relevant and supported by GBIF. We therefore had to create our own xsd using the proper eml namespace, 2.1.1 at that time. Having several schemas for the same namespace is nothing uncommon in the XML world.

It also specifies exactly what GBIF support in the additionalMetadata slot.

[timrobertson100]

Thanks, @mdoering. There was a reason we ended up hosting the full EML XSDs too though, and I forget exactly why. I seem to recall something like a broken EML build - ring any bells?

[mdoering]

I think it was this: https://github.com/gbif/rs.gbif.org/commit/0f0ca8423e09267fd515617dee018e008163edee

Use GBIF hosted xml.xsd as W3C times out

We slightly modified the original schemas to use a local xml.xsd file to avoid the timeouts

[timrobertson100]

Thanks @mdoering - there was something relating to EML XSDs before that (pre-github), but I forget the details now.

@pvgenuchten - is there anything more we can do here, or should we close the issue please?

[kmexter]

For the 2.2.0 EML schema, do you envision a similar implementation of the gbif profile?

FYI in fact we also have a question about moving over to 2.2. We are in the process of making our profile in EML 2.2 -- mapping from 2.1 is not such an issue, but adding the necessary semantic annotations will take us a bit longer -- and since some of our data ends up in GBIF, we were wondering about this. Would you not be able to harvest/recieve data that had eml.xml using eml 2.2?

[timrobertson100]

Would you not be able to harvest/recieve data that had eml.xml using eml 2.2?

My guess is that most likely we would not without some software changes. To be honest we've been neglecting this area mainly because there hasn't been much demand to change things and it's working out OK. If that demand came (e.g. a push from publishers) then we would re-prioritize of course.

[kmexter]

OK, I see. Makes sense. FYI there is a demand from various European FAIR data programmes that metadata are semantically annotated. This is possible in a machine-interoperable and standardised way in EML 2.2 but not fully so in EML 2.1. That is why we are updating, to become more FAIR.

[jdpye]

Hi gang, sorry to revive this thread but we had a set of workshops across Canada recently where we picked up DwC and EML and talked about making DwC-A for all sorts of data, and one big ask from our colleagues on the federal side was for a nice way to properly treat multilingual data. It looks like later EML versions allow for all fields to have a language attribute, and this would solve Canada's requirement to provide equal billing in both official languages, plus the obvious advantages to pan-European organizations.

Is it still the plan to update the GBIF EML schema, and if so, would there be interest in building up the IPT's awareness of this aspect, and potentially rolling all the way into field-by-field multilingual support for GBIF-EML?

[CecSve]

Would you not be able to harvest/recieve data that had eml.xml using eml 2.2?

My guess is that most likely we would not without some software changes. To be honest we've been neglecting this area mainly because there hasn't been much demand to change things and it's working out OK. If that demand came (e.g. a push from publishers) then we would re-prioritize of course.

We have had some issues come up on help desk the last six months with publishers using EML version 2.1.1 so there seem to be a need to support this version. There was a miscommunication between EML versions and GBIFs EML schema so please ignore this comment.

[kmexter]

Would you not be able to harvest/recieve data that had eml.xml using eml 2.2?

My guess is that most likely we would not without some software changes. To be honest we've been neglecting this area mainly because there hasn't been much demand to change things and it's working out OK. If that demand came (e.g. a push from publishers) then we would re-prioritize of course.

We have had some issues come up on help desk the last six months with publishers using EML version 2.1.1 so there seem to be a need to support this version.

:-D :-D :-D

[kmexter]

So that means still no plans to update to eml 2.2? If it is a question of human resources, we of the open science team at VLIZ can help out, as we have been investigating converting our 2.1s to 2.2s for a while...We are not implementing only because much of our data goes to GBif and we thus prefer to export an EML that GBif can work with. However, various issues on annotations (to make metadata more machine-accessible), the location of the physical-distribution module (which e.g. geonetwork does not work well with being placed in additionalMetadata), language (as @jdpye mentioned in a comment above, we in Belgium also have a multi-lingual audience), other catalogues we export to (which have updated to eml 2.2), keep coming up for us. Not that I am trying to push here, or anything :-D....just encourage and offer a hand

[timrobertson100]

Thank you very much @kmexter - It really is just a case that we haven't gotten to it... That you are keen to help would be greatly appreciated.

I think as a starting point, I suggest we document the changes from our (fairly minimal) profile to the latest version. We can then refer to that in the github issues for the code changes needed in the IPT and the Registry.

Would you be willing to try and start that perhaps?

[kmexter]

Sure, we can help: we have done a similar exercise here - quite a while ago so I will have to drag out the notes and refresh my memory, but I'd be happy to share that with you.

[timrobertson100]

Thank you very much

[kmexter]

you can email me directly on katrina.exter@vliz.be. just tell me what you need or we can have a chat. k

[mdoering]

I leave here a few notes on changes in EML 2.2 that I would like to see discussed as candidates for a GBIF implementation - apart from just a blind 1:1 upgrade. Especially the much improved support for bibliographic references using BibTex would help a lot to be more compatible to the metadata handled in ChecklistBank and ColDP.

• dataset/licensed instead of the rather unstructured way of handling licenses
• dataset/publisher to track publisher infos
• dataset/pubPlace see above
• dataset/literatureCited instead of dataset/additionalMetadata/gbif/bibliography
  • bibtex support would be highly welcome!
• dataset/referencePublication source reference for plazi articles? Might not be the right use of the element
• dataset/abstract/markdown COL and other datasets place markdown right into the abstract. Supporting the markdown element would be much better
• dataset/acknowledgements useful addition to dataset metadata? Not supported (yet) in CLB

[mdoering]

The official EML Data Paper Example seems useful to look at: https://github.com/NCEAS/eml/blob/main/src/test/resources/eml-data-paper.xml

[mike-podolskiy90]

I've created a new GBIF EML profile of EML 2.2 https://rs.gbif-uat.org/schema/eml-gbif-profile/1.3/eml.xsd It is now deployed to GBIF UAT environment, including IPT (ipt.gbif-uat.org) and the api.

Goal was to migrate with minimal changes, so any further improvements will be added incrementally.

[MattBlissett]

I've gone through the new features of EML 2.2.0, as well as older elements which weren't included in the GBIF Metadata Profile. I'll list them together, as it's over 10 years since the previous version so it's worth reconsidering everything. New in 2.2.0 is marked with ¤.

EML 2.2.0: https://eml.ecoinformatics.org/whats-new-in-eml-2-2-0.html

EML 2.1.1: https://sbclter.msi.ucsb.edu/external/InformationManagement/EML_211_schema/docs/eml-2.1.1/

GBIF Metadata Profile: https://ipt.gbif.org/manual/en/ipt/latest/gbif-metadata-profile

New or absent elements.

Those I've put in bold seem good contenders for GBIF's support.

• dataset/shortname — unlikely we'd use this
• dataset/title — we describe the older multilingual method in the GMP document, but never implemented it
• dataset/creator/individualName/salutation — not so much for EML, but could be useful for some publishers
• dataset/series — don't think this is relevant to us
• dataset/licensed/url — ¤ replaces the ulink kludge we have in dataset/intellectualRights. EML also recommend a particular set of licence URIs, different to what we use ourselves (see also https://github.com/gbif/ipt/issues/1967)
• dataset/distribution/online — could describe endpoints, or locations for downloads, rather than only homepages. Should replace additionalMetadata/metadata/gbif/physical/....
• dataset/coverage/taxonomicCoverage/taxonomicClassification/taxonId — ¤ taxonomic identifiers now supported.
• dataset/annotation — ¤ structured annotations (key-value).
• dataset/purpose — "A synopsis of the purpose of this dataset"
• dataset/introduction — ¤ recommended for data papers
• dataset/gettingStarted — ¤ data papers
• dataset/acknowledgements — ¤ also recommended for data papers
• dataset/publisher — for the publisher?
• dataset/pubPlace ­— just a publisher address, not sure why this wouldn't be in dataset/publisher/address. Maybe in case it's different?
• dataset/project/award — ¤ new for structured funding information, all useful for project tracking
• dataset/dataTable ­— data, could hold data mappings for published data and/or downloads
• dataset/referencePublication — ¤ "Common cases where a Reference Publication may be useful include when a data paper is published that describes the dataset, or when a paper is intended to be the canonical or examplar reference to the dataset." Could reference all datasets in a download, and source in treatments.
• dataset/usageCitation — ¤ would be a way to show citations of GBIF datasets. Can now use BibTeX.
• dataset/literatureCited — ¤ replaces additionalMetadata/metadata/gbif/bibliography. Can now use BibTeX.

Multilingual support

We describe this in the GMP guide, but never implemented it in the Registry. Now we can implement the newer method.

This seems important for a new profile version, but needs less-compatible API changes to the Registry to expose the result. Perhaps we could implement something without changing /v1/dataset (e.g. return English, or use the Accept-Language header, or use the primary language of the dataset), use /v1/dataset/..../document as a way to expose the full details, giving time to decide upon /v2/dataset.

The existing additionalMetadata/metadata/gbif/metadataLanguage should be using the xml:lang attribute on the eml or dataset element.

Markdown

Markdown in supported in several places. References to images are supported.

The current IPT and Registry implementations are using inlined, escaped HTML, though the EML schema says it should be DocBook elements. Changing to the DocBook elements might help interoperability with other publishers of EML, assuming they do things properly.

This seems important for a new profile version, but also needs less-compatible API changes to the Registry.

BibTeX

Citations can be provided in BibTeX as well as as plain text strings.

GBIF Extension

These are all the elements of the GBIF extension:

• dateStamp — could perhaps be replaced with dataset/maintenance/changeHistory/changeDate, and at the same time add version information. (Someone has asked for that, I can't find the issue.)
• metadataLanguage — could use xml:lang on dataset instead
• heirarchyLevel — keep
• citation ­— keep, EML suggest generating a citation from the components (authors etc)
• bibliography — replace with dataset/literatureCited
• physical — move to dataset/physical
• resourceLogoUrl — NCD elements. Should these all go in the new Semantic Metadata section? (And be replace with TDWG Collection Descriptions elements?)
• collection/parentCollectionIdentifier
• collection/collectionName
• collection/collectionIdentifier
• formationPeriod
• livingTimePeriod
• specimenPreservationMethod
• jgtiCuratorialUnit
• replaces — keep?

Other observations

We should check if we can support a full EML document (with the elements not in our profile), so we can support VLIZ providing a more complete document (as above).

[kmexter]

So we have been working here on creating a 2.2 profile based on our 2.1 profile. Since I could not attach an eml file to this comment, I have put it here instead https://drive.google.com/file/d/1GpxdscHCDappsk3GUMU4cmaRyt1AK_Ch/view?usp=sharing

What we have added to make our 2.2 are the following

• referenceCitations for what is our database keyword "describing a dataset"
• usageCitation for "based on this dataset"
• literatureCited for "used in this dataset"
• we added the \<annotation> element to the \<associatedParty> element to indicate what their role is in way that adds semantics (tho we are struggling to find any vocabulary that defines people roles!)
• we realised that if you have 3 keywords, from 2 vocabs and one from no vocab, then you need 3 \<keywordSet>s: for keyword1-vocab1, keyword2-vocab2, keyword3 (that was a mistake of ours from before)
• Indeed also used the \<url> in \<licensed> to point to a web-resolvable URL for the licence (now we just need someone to create a licence for "under embargo" and for "contact for access"!)
• we adeed id= to the \<geographicalCoverage> and then used \<annotation> in order to be able to semantically annotate the geographical coverage, i.e. we put the URL to the MarineRegion (as we are a marine institute), that is included in the geographical coverage
• we used the new taxonId for the taxonomic coverage, but it was unclear to us whether to put only the AphiaID in there, or the full AphiaID address (i.e. do we use a number or a URL). Our interpretation of the (could be better) EML explanation was that it should just be an ID, so we then added \<annotation> in order to do the semantics, i.e. add the fully URL of the AphiaID, In this way, we realised you could maybe add multiple IDs, tho that is probably not such a good idea, thinking about it....too confusing
• we used \<methodStep> to add platforms (actually, this is also valid in 2.1: we added this because it is required by a project we are exposing our metadata to)
• We moved \<physical> out of \<additionalMetadata> and into because we think this is where it is supposed to go (according to the EML schema) --> this is the same as what Matt is suggesting above, although we'd have to sort out the syntax a bit (do you need \<physical> or just \<distribution>?)
• We are playing with using \<annotation> to link to related datasets. We cannot see that you can do this otherwise in EML, but what you can do is add to \<dataset> some \<annotations> in which we added specific references to the other metadata records that are "a parent of" or "related to" the dataset in this metadata record. Unclear if we will adopt this because it is a bit of a hack - wondering what GBIF thinks about this?
• we added \<parameter> to \<additionalMetadata> because a project we are part of required that we add parametres in the dataset, and there is no where else to do that

We were thinking of asking EML to add some new elements in their next version - would be interesting to know if you also think these are important

• parameters (like keywords) or parameterCoverage (to go in coverage)
• instruments and platforms: we know you can add those in \<methodStep> but really \<methodStep> is supposed to be used in a block to describe things you did and we are not sure people will want to add quite that much information to their metadata: it seems logical to us that you have a section for instruments/platforms so that these can form part of the discovery metadata
• to have roles be semantically annotatable (rather than using \<annotation> to do that)
• add a permits element (e.g. especially important for being able to transmit information on ABS permits, this being a bit of a politcal hot potato)

While waiting for EML to respond to any request that I send, I was wondering whether GBIF would like to consider any of these elements I list above, as in adding them in \<additionalMetadata>, creating your own elements?