search

metno / S-ENDA-documentation

Temporary documentation and use case descriptions for the S-ENDA project - when concepts are tested and verified, content is gradually moved to more long term solutions.
https://s-enda-documentation.readthedocs.io/
2 stars 6 forks source link

parent dataset not found on production #307

Closed TAlonglong closed 1 year ago

TAlonglong commented 1 year ago

A parent datasat has ID c7f8731b-5cfe-4cb5-ac57-168a19a2957b

This can be found at dev

https://csw.s-enda-dev.k8s.met.no/?service=CSW&version=2.0.2&request=GetRepositoryItem&id=c7f8731b-5cfe-4cb5-ac57-168a19a2957b

Staging

https://csw.s-enda-staging.k8s.met.no/?service=CSW&version=2.0.2&request=GetRepositoryItem&id=c7f8731b-5cfe-4cb5-ac57-168a19a2957b

but not production https://csw.s-enda.k8s.met.no/?service=CSW&version=2.0.2&request=GetRepositoryItem&id=c7f8731b-5cfe-4cb5-ac57-168a19a2957b

TAlonglong commented 1 year ago

But if I try to ingest the parent in question I get:

remotesensing@ppi-r8login-b1:~/mmd-xml-files$ curl --data-binary @noaa20-viirs-parent.xml https://dmci.s-enda.k8s.met.no/v1/insert
The following distributors failed: file, pycsw
 - file: File already exists: c7f8731b-5cfe-4cb5-ac57-168a19a2957b.xml
 - pycsw: <?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- pycsw 2.7.dev0 -->
<ows:ExceptionReport xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:dct="http://purl.org/dc/terms/" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:gml="http://www.opengis.net/gml" xmlns:ows="http://www.opengis.net/ows" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.2.0" language="en-US" xsi:schemaLocation="http://www.opengis.net/ows http://schemas.opengis.net/ows/1.0.0/owsExceptionReport.xsd"><ows:Exception exceptionCode="NoApplicableCode" locator="insert"><ows:ExceptionText>Transaction (insert) failed: (psycopg2.errors.NotNullViolation) null value in column "metadata_type" of relation "records" violates not-null constraint
DETAIL:  Failing row contains (c7f8731b-5cfe-4cb5-ac57-168a19a2957b, gmd:MD_Metadata, http://www.isotc211.org/2005/gmd, local, 2023-03-08T12:59:12Z, &lt;gmd:MD_Metadata xmlns:gmd="http://www.isotc211.org/2005/gmd" xm..., c7f8731b-5cfe-4cb5-ac57-168a19a2957b English series_time Norwegi..., null, null, null, series, Direct Broadcast data processed in satellite swath to L1C., null, Direct Broadcast data received at MET NORWAY Oslo. Processed by ..., Earth Science &gt; Atmosphere &gt; Atmospheric radiation,Government Ag..., null, null, null, 2020-05-05T22:39:34.155886Z, null, climatologyMeteorologyAtmosphere, eng, null, null, null, Norwegian Meteorological Institute, null, otherRestrictions, null, 2020-05-07, null, null, 2020-05-07, 2020-05-07, null, null, EPSG:4326, null, null, null, null, POLYGON((-180.00 25.00, -180.00 90.00, 180.00 90.00, 180.00 25.0..., , null, null, null, null, null, null, null, null, null, null, pointOfContact, null, null, null, null, null, null, null, null, null, '-01':151,168 '-02':124,150 '-04':140 '-05':18,44,201,202 '-06':..., 0103000020E6100000010000000500000000000000008066C000000000000039...).

[SQL: INSERT INTO records (identifier, typename, schema, mdsource, insert_date, xml, anytext, metadata, metadata_type, language, type, title, title_alternate, abstract, keywords, keywordstype, parentidentifier, relation, time_begin, time_end, topicategory, resourcelanguage, creator, publisher, contributor, organization, securityconstraints, accessconstraints, otherconstraints, date, date_revision, date_creation, date_publication, date_modified, format, source, crs, geodescode, denominator, distancevalue, distanceuom, wkt_geometry, servicetype, servicetypeversion, operation, couplingtype, operateson, operatesonidentifier, operatesoname, degree, classification, conditionapplyingtoaccessanduse, lineage, responsiblepartyrole, specificationtitle, specificationdate, specificationdatetype, platform, instrument, sensortype, cloudcover, bands, links, anytext_tsvector, wkb_geometry) VALUES (%(identifier)s, %(typename)s, %(schema)s, %(mdsource)s, %(insert_date)s, %(xml)s, %(anytext)s, %(metadata)s, %(metadata_type)s, %(language)s, %(type)s, %(title)s, %(title_alternate)s, %(abstract)s, %(keywords)s, %(keywordstype)s, %(parentidentifier)s, %(relation)s, %(time_begin)s, %(time_end)s, %(topicategory)s, %(resourcelanguage)s, %(creator)s, %(publisher)s, %(contributor)s, %(organization)s, %(securityconstraints)s, %(accessconstraints)s, %(otherconstraints)s, %(date)s, %(date_revision)s, %(date_creation)s, %(date_publication)s, %(date_modified)s, %(format)s, %(source)s, %(crs)s, %(geodescode)s, %(denominator)s, %(distancevalue)s, %(distanceuom)s, %(wkt_geometry)s, %(servicetype)s, %(servicetypeversion)s, %(operation)s, %(couplingtype)s, %(operateson)s, %(operatesonidentifier)s, %(operatesoname)s, %(degree)s, %(classification)s, %(conditionapplyingtoaccessanduse)s, %(lineage)s, %(responsiblepartyrole)s, %(specificationtitle)s, %(specificationdate)s, %(specificationdatetype)s, %(platform)s, %(instrument)s, %(sensortype)s, %(cloudcover)s, %(bands)s, %(links)s, %(anytext_tsvector)s, %(wkb_geometry)s)]
[parameters: {'identifier': 'c7f8731b-5cfe-4cb5-ac57-168a19a2957b', 'typename': 'gmd:MD_Metadata', 'schema': 'http://www.isotc211.org/2005/gmd', 'mdsource': 'local', 'insert_date': '2023-03-08T12:59:12Z', 'xml': '&lt;gmd:MD_Metadata xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/19 ... (14880 characters truncated) ... t has been provided&lt;/gco:CharacterString&gt;&lt;/gmd:statement&gt;&lt;/gmd:LI_Lineage&gt;&lt;/gmd:lineage&gt;&lt;/gmd:DQ_DataQuality&gt;&lt;/gmd:dataQualityInfo&gt;&lt;/gmd:MD_Metadata&gt;', 'anytext': 'c7f8731b-5cfe-4cb5-ac57-168a19a2957b English series_time Norwegian Meteorological Institute Meteorologisk institutt post@met.no pointOfContact 2020-0 ... (1714 characters truncated) ... ets and services 2010-12-08 publication The dataset has not been evaluated against the requirements of Inspire No lineage statement has been provided', 'metadata': None, 'metadata_type': None, 'language': None, 'type': 'series', 'title': 'Direct Broadcast data processed in satellite swath to L1C.', 'title_alternate': None, 'abstract': 'Direct Broadcast data received at MET NORWAY Oslo. Processed by standard processing software to geolocated and calibrated values in satellite swath in received instrument resolution.', 'keywords': 'Earth Science &gt; Atmosphere &gt; Atmospheric radiation,Government Agencies-non-US &gt; Norway &gt; NO/MET &gt; Norwegian Meteorological Institute,Geographic Region &gt; Northern Hemisphere,Meteorological geographical features,Atmospheric conditions,Oceanographic geographical features,Weather and climate', 'keywordstype': None, 'parentidentifier': None, 'relation': None, 'time_begin': '2020-05-05T22:39:34.155886Z', 'time_end': None, 'topicategory': 'climatologyMeteorologyAtmosphere', 'resourcelanguage': 'eng', 'creator': None, 'publisher': None, 'contributor': None, 'organization': 'Norwegian Meteorological Institute', 'securityconstraints': None, 'accessconstraints': 'otherRestrictions', 'otherconstraints': None, 'date': '2020-05-07', 'date_revision': None, 'date_creation': None, 'date_publication': '2020-05-07', 'date_modified': '2020-05-07', 'format': None, 'source': None, 'crs': 'EPSG:4326', 'geodescode': None, 'denominator': None, 'distancevalue': None, 'distanceuom': None, 'wkt_geometry': 'POLYGON((-180.00 25.00, -180.00 90.00, 180.00 90.00, 180.00 25.00, -180.00 25.00))', 'servicetype': '', 'servicetypeversion': None, 'operation': None, 'couplingtype': None, 'operateson': None, 'operatesonidentifier': None, 'operatesoname': None, 'degree': None, 'classification': None, 'conditionapplyingtoaccessanduse': None, 'lineage': None, 'responsiblepartyrole': 'pointOfContact', 'specificationtitle': None, 'specificationdate': None, 'specificationdatetype': None, 'platform': None, 'instrument': None, 'sensortype': None, 'cloudcover': None, 'bands': None, 'links': None, 'anytext_tsvector': None, 'wkb_geometry': None}]
(Background on this error at: http://sqlalche.me/e/13/gkpj).</ows:ExceptionText></ows:Exception></ows:ExceptionReport>
magnarem commented 1 year ago

This error are related to the postgres database schema differences between the dev-environment and the staging/prod -e environments.

senda-csw-sql $ diff -u s-enda.sql s-enda-dev.sql 
--- s-enda.sql  2023-03-28 14:08:51.782983783 +0200
+++ s-enda-dev.sql  2023-03-28 14:08:56.178975951 +0200
@@ -547,8 +547,6 @@
     insert_date text NOT NULL,
     xml character varying NOT NULL,
     anytext text NOT NULL,
-    metadata character varying,
-    metadata_type text NOT NULL,
     language text,
     type text,
     title text,
@@ -1157,10 +1155,10 @@

Lets hope this can be closed after clearing the databases in staging/ prod, and after we have rebuilt the catalog

johtoblan commented 1 year ago

This is the same as #130 in dmci