search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

AATAMS SATTAG - schemas - to keep or to delete ? #468

Closed lbesnard closed 2 years ago

lbesnard commented 3 years ago

The following legacy schemas

have new respective schemas:

1) Should both legacy ones be kept? If yes why? 2) Should the SQL reporting queries (https://github.com/aodn/reporting/blob/master/report/SQL_reporting_queries/aatams_sattag.sql) be modified accordingly?

evacougnon commented 3 years ago

Unfortunately, the 2 new schemas do not replace each of the old schemas.

At this stage we have to keep aatams_sattag_dm as we still have a dataset collection available on the Portal. This collection is different from aatams_sattag_qc_dm as it includes historical data.

aatams_sattag_nrt can probably be decommissioned, to be discussed with @ggalibert and @atkinsn . Earlier this year we decommissioned the aatams_sattag_nrt datatset collection from the portal as well as the pipeline. @atkinsn kept (and updated accordingly) the metadata records. I don't know what's the procedure about decommissioning a schema but I guess we do not use the data anymore. Same about reporting, I don't think we have anything to report about aatams_sattag_nrt.

lbesnard commented 2 years ago

happy to investigate if this aatams_sattag_nrt can be decomissioned

lbesnard commented 2 years ago

So unfortunately, the aatams_sattag_dm schema relies on the aatams_sattag_dm schema. There is an interdependence which would break the the aatams_sattag_dm harvester. see https://github.com/aodn/chef-private/blob/master/data_bags/talend/aatams_sattag_dm.json#L5

The aatams_sattag_dm schema cannot be removed unless both collections (NRT and DM) can be decommissioned