search

wmo-im / pywcmp

pywcmp provides validation and quality assessment capabilities for the WMO WIS Core Metadata Profile (WCMP)
https://community.wmo.int/activity-areas/wis
Other
9 stars 8 forks source link

KPI: increase error robustness of KPI checks #50

Closed antje-s closed 3 years ago

antje-s commented 3 years ago

Dear pywmcp team,

the KPI checks are running fine. I just noticed that in case of problems with single KPI check the script aborts completely. It would be good to jump to the next KPI check in case of problems, so that all KPI checks are tried for each metadata record.

Examples: 1) Links check failed: urn%3Ax-wmo%3Amd%3Aint.wmo.wis%3A%3ACSCZ10OKPR.xml --> [2021-05-31T06:38:56Z] INFO - Running KPI-8: Links health [2021-05-31T06:38:56Z] INFO - "http://wis.wmo.int/2012/codelists/WMOCodeLists.xml#WMO_GTSProductCategoryCode" cannot be resolved! --> not checked KPI-9 / KPI-10 / KPI-11 / KPI-12

2) undefined namespace prefix urn%3Ax-wmo%3Amd%3Aint.eumetsat%3AEO%3AEUM%3ADAT%3ASENTINEL-3%3ASR_1_SRA_A_NTC.xml --> [2021-05-31T07:21:37Z] INFO - Running KPI-7: Graphic overview for non bulletins metadata records --> not checked KPI-8 / KPI-9 / KPI-10 / KPI-11 / KPI-12 call via command line for single xml returned: ... File "...python3.6/site-packages/pywcmp-0.3.dev0-py3.6.egg/pywcmp/kpi.py", line 962, in evaluate result = getattr(self, kpi)() File "...python3.6/site-packages/pywcmp-0.3.dev0-py3.6.egg/pywcmp/kpi.py", line 531, in kpi_007 graphic_overviews = self.exml.xpath(xpath, namespaces=self.namespaces) File "src/lxml/etree.pyx", line 2296, in lxml.etree._ElementTree.xpath File "src/lxml/xpath.pxi", line 357, in lxml.etree.XPathDocumentEvaluator.call__ File "src/lxml/xpath.pxi", line 225, in lxml.etree._XPathEvaluatorBase._handle_result lxml.etree.XPathEvalError: Undefined namespace prefix

Kind regards Antje

tomkralidis commented 3 years ago

Thanks @antje-s. @josusky has fixed in this master. Are you able to update your install based on latest master and try again?

antje-s commented 3 years ago

Many thanks @josusky and @tomkralidis No problem I will do that and get back to you

antje-s commented 3 years ago

One step further, it continues to run even with unsuccessful link check, but there is still a problem with KPI 11 e.g. urn:x-wmo:md:int.eumetsat:EO:EUM:DAT:SENTINEL-3:SR_1_SRA_A___NTC.xml now stopped at [2021-06-04T06:28:09Z] INFO - Running KPI-11: Codelists validation ... File "/home/wis/myScripts/venv/pywcmp/lib64/python3.6/site-packages/pywcmp-0.3.dev0-py3.6.egg/pywcmp/kpi.py", line 851, in kpi_011 if xpath2.text in self.codelists[key][codelist]: KeyError: 'MD_DistributionScopeCode' ...

josusky commented 3 years ago

OK, I have added one more exception handling into KPI-11, so now it will report: "Line 377: Invalid code list reference: 'MD_DistributionScopeCode' is not defined in 'wmo'" and keep going.

antje-s commented 3 years ago

Thanks @josusky, I updated my repository with the latest development version from source and start a new run

antje-s commented 3 years ago

...looks good, now runs completely through - thanks again