Improve error message

[mortenwh]

https://github.com/metno/mmd_agent/blob/d11e9a8d57aa3f4e6a6448bf1e322af4556de751/mmd_agent/agent.py#L90

Should be: "Failed to push MMD file %s to DMCI" % filename

[myggen]

I think it should be "Failed to push MMD file %s to DMCI http status code: %s " % (filename, http_status_code)

[mortenwh]

I think it should be "Failed to push MMD file %s to DMCI http status code: %s " % (filename, http_status_code)

Yes, that would be great. But maybe even "Failed to push MMD file %s to DMCI.\nHTTP status code: %s" % (filename, http_status_code)

[myggen]

I think that newline makes it harder to grep the logs

[mortenwh]

I think that newline makes it harder to grep the logs

Aha, ok. What about "Failed to push MMD file %s to DMCI. HTTP status code: %s." % (filename, http_status_code)?

[myggen]

Looks good to me

[shamlymajeed]

now the failed to save log is like this.

__main__:2024-01-04 16:52:10,930:ERROR:Failed to save
__main__:2024-01-04 16:52:10,931:ERROR:500,The following distributors failed: pycsw, solr
 - pycsw: <?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- pycsw 2.7.dev0 -->
<ows:ExceptionReport xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:dct="http://purl.org/dc/terms/" xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:gml="http://www.opengis.net/gml" xmlns:ows="http://www.opengis.net/ows" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.2.0" language="en-US" xsi:schemaLocation="http://www.opengis.net/ows http://schemas.opengis.net/ows/1.0.0/owsExceptionReport.xsd"><ows:Exception exceptionCode="NoApplicableCode" locator="insert"><ows:ExceptionText>Transaction (insert) failed: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "records_pkey"
DETAIL:  Key (identifier)=(no.met.dev:30c74543-afe9-4025-9be1-9faeab0e972d) already exists.

[SQL: INSERT INTO records (identifier, typename, schema, mdsource, insert_date, xml, anytext, language, type, title, title_alternate, abstract, keywords, keywordstype, parentidentifier, relation, time_begin, time_end, topicategory, resourcelanguage, creator, publisher, contributor, organization, securityconstraints, accessconstraints, otherconstraints, date, date_revision, date_creation, date_publication, date_modified, format, source, crs, geodescode, denominator, distancevalue, distanceuom, wkt_geometry, servicetype, servicetypeversion, operation, couplingtype, operateson, operatesonidentifier, operatesoname, degree, classification, conditionapplyingtoaccessanduse, lineage, responsiblepartyrole, specificationtitle, specificationdate, specificationdatetype, platform, instrument, sensortype, cloudcover, bands, links, anytext_tsvector, wkb_geometry) VALUES (%(identifier)s, %(typename)s, %(schema)s, %(mdsource)s, %(insert_date)s, %(xml)s, %(anytext)s, %(language)s, %(type)s, %(title)s, %(title_alternate)s, %(abstract)s, %(keywords)s, %(keywordstype)s, %(parentidentifier)s, %(relation)s, %(time_begin)s, %(time_end)s, %(topicategory)s, %(resourcelanguage)s, %(creator)s, %(publisher)s, %(contributor)s, %(organization)s, %(securityconstraints)s, %(accessconstraints)s, %(otherconstraints)s, %(date)s, %(date_revision)s, %(date_creation)s, %(date_publication)s, %(date_modified)s, %(format)s, %(source)s, %(crs)s, %(geodescode)s, %(denominator)s, %(distancevalue)s, %(distanceuom)s, %(wkt_geometry)s, %(servicetype)s, %(servicetypeversion)s, %(operation)s, %(couplingtype)s, %(operateson)s, %(operatesonidentifier)s, %(operatesoname)s, %(degree)s, %(classification)s, %(conditionapplyingtoaccessanduse)s, %(lineage)s, %(responsiblepartyrole)s, %(specificationtitle)s, %(specificationdate)s, %(specificationdatetype)s, %(platform)s, %(instrument)s, %(sensortype)s, %(cloudcover)s, %(bands)s, %(links)s, %(anytext_tsvector)s, %(wkb_geometry)s)]
[parameters: {'identifier': 'no.met.dev:30c74543-afe9-4025-9be1-9faeab0e972d', 'typename': 'gmd:MD_Metadata', 'schema': 'http://www.isotc211.org/2005/gmd', 'mdsource': 'local', 'insert_date': '2024-01-04T16:52:10Z', 'xml': '&lt;gmd:MD_Metadata xmlns:gmd="http://www.isotc211.org/2005/gmd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/19 ... (27210 characters truncated) ... t has been provided&lt;/gco:CharacterString&gt;&lt;/gmd:statement&gt;&lt;/gmd:LI_Lineage&gt;&lt;/gmd:lineage&gt;&lt;/gmd:DQ_DataQuality&gt;&lt;/gmd:dataQualityInfo&gt;&lt;/gmd:MD_Metadata&gt;', 'anytext': 'no.met.dev:30c74543-afe9-4025-9be1-9faeab0e972d English no.met.dev:e2228b20-e62f-436b-aa17-7716acfe8e3a Norwegian Meteorological Institute Meteorolog ... (6863 characters truncated) ... ets and services 2010-12-08 publication The dataset has not been evaluated against the requirements of Inspire No lineage statement has been provided', 'language': None, 'type': 'dataset', 'title': 'NOAA-21 VIIRS Day Night Band(750m spatial resolution) level-1C direct broadcast data in satellite swath, 2024-01-04T14:12:03Z.', 'title_alternate': None, 'abstract': 'Direct Broadcast satellite data received at MET NORWAY Oslo from the satellite NOAA-21 instrument VIIRS. Data is from 2024-01-04T14:12:03Z to 2024-01 ... (627 characters truncated) ... n of 375m. The (D)ay (N)ight (B)and is a broadspectrum visible channel sensitive to both refelected sun light at day and all visible lights at night.', 'keywords': 'Earth Science &gt; Atmosphere &gt; Atmospheric radiation,Government Agencies-non-US &gt; Norway &gt; NO/MET &gt; Norwegian Meteorological Institute,Geographic Region &gt; Northern Hemisphere,Meteorological geographical features,Atmospheric conditions,Oceanographic geographical features,Weather and climate', 'keywordstype': None, 'parentidentifier': 'no.met.dev:e2228b20-e62f-436b-aa17-7716acfe8e3a', 'relation': None, 'time_begin': '2024-01-04T14:12:03.739788Z', 'time_end': '2024-01-04T14:22:00.270950Z', 'topicategory': 'climatologyMeteorologyAtmosphere', 'resourcelanguage': 'eng', 'creator': None, 'publisher': None, 'contributor': None, 'organization': 'Norwegian Meteorological Institute', 'securityconstraints': None, 'accessconstraints': 'otherRestrictions', 'otherconstraints': None, 'date': '2024-01-04', 'date_revision': None, 'date_creation': None, 'date_publication': '2024-01-04T14:28:30Z', 'date_modified': '2024-01-04', 'format': 'NetCDF-CF', 'source': None, 'crs': 'EPSG:4326', 'geodescode': None, 'denominator': None, 'distancevalue': None, 'distanceuom': None, 'wkt_geometry': 'POLYGON((-73.17 36.54, -73.17 76.48, 7.12 76.48, 7.12 36.54, -73.17 36.54))', 'servicetype': '', 'servicetypeversion': None, 'operation': None, 'couplingtype': None, 'operateson': None, 'operatesonidentifier': None, 'operatesoname': None, 'degree': None, 'classification': None, 'conditionapplyingtoaccessanduse': None, 'lineage': None, 'responsiblepartyrole': 'pointOfContact', 'specificationtitle': None, 'specificationdate': None, 'specificationdatetype': None, 'platform': None, 'instrument': None, 'sensortype': None, 'cloudcover': None, 'bands': None, 'links': '[{"name": null, "description": "Open-source Project for a Network Data Access Protocol", "protocol": "OPENDAP:OPENDAP", "url": "https://thredds.met.n ... (546 characters truncated) ... 2024/01/04/noaa21-viirs-dnb-20240104141203-20240104142200.nc?service=WMS&amp;version=1.3.0&amp;request=GetCapabilities?SERVICE=WMS&amp;REQUEST=GetCapabilities"}]', 'anytext_tsvector': None, 'wkb_geometry': None}]
(Background on this error at: http://sqlalche.me/e/13/gkpj).</ows:ExceptionText></ows:Exception></ows:ExceptionReport>
 - solr: Document already exists in index, no.met.dev:30c74543-afe9-4025-9be1-9faeab0e972d

The expected log is

 Failed to push MMD file noaa21-viirs-dnb-20240104141203-20240104142200.nc to DMCI. HTTP status code: 500.

?

[mortenwh]

It was the first line that I wanted to change. I'm not sure about the rest, although it could be useful. Maybe: Failed to push MMD file noaa21-viirs-dnb-20240104141203-20240104142200.nc (no.met.dev:30c74543-afe9-4025-9be1-9faeab0e972d) to DMCI. HTTP status code: 500. The following distributors failed: pycsw, solr.

It would also be useful to know the problem, in this case "already exists" for both pycsw and solr. Can you pick up the text following DETAIL from pycsw, and the solr text? Then it should be printed on the same line, because what @myggen pointed out.

Are these files stored anywhere?

[shamlymajeed]

The second line is actually the response status code and text we are getting from dmci. I can try to extract the text following 'Details.'

These files actually exists . mmd agent is now in jetstream mode.So its just verifying whether all the messages received are processed. :)