search

lter / LTER-core-metabase

RDB model, based on the GCE LTER Metabase, with adaptations from other LTER sites
https://lter.github.io/LTER-core-metabase/
GNU General Public License v3.0
9 stars 4 forks source link

unit list should be modified to contain list and attributes from EML 2.2 #75

Open mobb opened 4 years ago

mobb commented 4 years ago

the list of units that were allowed appear as a has been updated in EML 2.2. core-metabase should upgrade to this list The changes are summarized here: https://github.com/NCEAS/eml/issues/289#issuecomment-377335643

gastil commented 4 years ago

I very much look forward to using EML2.2 but Version 1 lter-core-mb is for EML2.1.

I know the EML2.2 standardUnit list is much better, and I welcome that. But I also want to be sure we will be able to bulk upload EML2MB, which is expected to be entirely EML2.1, at least initially. I would not want the units to make an EML2MB upload fail. Do we need a way to specify the EML version, or a configuration to use the old or new units table? I think MCR is immune, using entirely customUnits already. But I want to offer bulk EML2MB for other sites too.

I see EML2MB as a 3-step process: (1) load it all as-is (2) verify the original EML can be re-generated, then (3) refine the metadata with upgrades such as units, harmonization, etc. To accomplish this, mb needs to be able to hold EML2.1.

gastil commented 4 years ago

Summary of discussion today with Margaret:

The standard units in EML2.2 have two new columns: deprecated in favor of and synonym for udunits. So we need to add 2 columns to the units table.

Any unit from EML2.1 will still be accepted by the EML2.2 parser. But for a unit now listed as deprecated in favor of we should use UPDATE the cross reference table to choose the better unit instead. I think we can do that with an sql script.

Migrating to the new EML2.2 units, that is, loading them into the mb table, could be tricky. Easier if we have UPSERT (insert or if preexisting, update). Maybe load the 2.2 table in scratch then populate new rows with a select that omits pre-existing rows.

The good news is the new units will not prevent us from uploading old units in bulk EML2MB.