search

OpenWIS / openwis

http://openwis.github.io/openwis
GNU General Public License v3.0
11 stars 15 forks source link

Metadata Synchronisation - issue #264

Open woollattd opened 6 years ago

woollattd commented 6 years ago

It appears that there is an issue synchronising with Meteo France via OpenWIS (we're on v3.14.5). Currently Exeter and Melbourne are showing 4760 records for Toulouse - when Toulouse shows there should be 4773. (difference = 13) It has been like this for a few weeks.

The outstanding files appear to be: urn:x-wmo:md:fr.meteo.dcpc-nwp::HYCOM2D-MARO_0.04_SP1_GRIB2;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::HYCOM2D-MARO_0.04_SP1_TXT;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::HYCOM2D-MARP_0.04_SP1_GRIB2;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::HYCOM2D-MARP_0.04_SP1_TXT;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::HYCOM2D-WARO_0.04_SP1_GRIB2;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::HYCOM2D-WARO_0.04_SP1_TXT;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::HYCOM2D-WARP_0.04_SP1_GRIB2;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::HYCOM2D-WARP_0.04_SP1_TXT;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::MFWAM_0.025_SP1_GRIB2;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::MFWAM_0.1_SP1_GRIB2;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::MFWAM_0.5_SP1_GRIB2;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::WW3-MARP_0.01_SP1_NC;0; urn:x-wmo:md:fr.meteo.dcpc-nwp::WW3-WARP_0.01_SP1_NC;0;

These appear to be marked as 'Not Compliant' in a full re-synch (how did Toulouse get them into OpenWIS? Maybe we do the same?) There are also 6 that were deleted but hadn't been before - so is OpenWIS not spotting these properly during Synchronisation?

report

woollattd commented 6 years ago

It also appears be an issue generally with non-OpenWIS GISCs as some of them are shoiwing incorrect totals for Toulouse. eg DWD has 3534 records CMA has 4500

Of course it may be that metadata synchronisation just doesn't work properly in general as it seems that it's not just Toulouse records that are out of synch between all the GISCs, but I would hope that it would work between GISCs using OpenWIS.

benjaminsaclier commented 6 years ago

Hi Dom,

Can you please provide us the error which is log in metadata.log file when you try to harvest these metadata ? Also can you attach your "openwis-metadataportal.properties" configuration file

Thanks,

Ben

woollattd commented 6 years ago

Hi

Hope all went well last week. Just catching up now I’m back in the office.

We are having big problem with our production system at the moment. I am waiting for it to be rebuilt, seems to be taking longer than expected. Once that is done we can take a look at this issue.

I am also still waiting for 3.14.9 to be installed on test so I can check that out too. But I have been waiting over a year for this to be done (originally asked for 3.14.8)

Cheers Dom Dominic Woollatt | MSS Maintenance & Support Met Office ® E2-091 Fitzroy Road Exeter EX1 3PB United Kingdom Tel: +44 (0)1392 886059 Web: http://www.metoffice.gov.uk<blocked::http://www.metoffice.gov.uk/>

woollattd commented 6 years ago

I am no longer able to extract the logs myself any more but will try and get someone to do so.

in the meantime here is an extract from the metadata.log for one entry it is now the same for 30 records it appears there is no GTS Category thus making it invalid.

2018-07-06 13:32:59,775 INFO [openwis.metadataaligner] - Creating local metadata for identifier: urn:x-wmo:md:fr.meteo.dcpc-eer::perlarp_locreg_1-4km_forecast_pollutant_0H_24H_30min1h 2018-07-06 13:33:00,069 INFO [openwis.extract.productmetadata] - Extracted FNC Pattern: 2018-07-06 13:33:00,069 INFO [openwis.extract.productmetadata] - Extracted Originator: NMC FRANCE - Météo-France 2018-07-06 13:33:00,069 INFO [openwis.extract.productmetadata] - Extracted Title: Forecasts of pollutant plume dispersion computed with the atmospheric model PERLE. Meteorological forcing: MesoNH coupled with ARPEGE for ecasts. 2018-07-06 13:33:00,069 INFO [openwis.extract.productmetadata] - Extracted Local Data Source: 2018-07-06 13:33:00,070 INFO [openwis.extract.productmetadata] - Ignored file extension: SHP 2018-07-06 13:33:00,070 INFO [openwis.extract.productmetadata] - Possible value for GTS Category and Data Policy:

2018-07-06 13:33:00,070 WARN [openwis.extract.productmetadata] - Product metadata is not valid. No GTS category. 2018-07-06 13:33:00,070 WARN [openwis.metadataaligner] - Unable to extract OpenWIS information for metadata: urn:x-wmo:md:fr.meteo.dcpc-eer::perlarp_locreg_1-4km_forecast_pollutant_0H_24H_30min1h

solson-nws commented 2 years ago

@yvesgoupil @davidpodeur -- If there is no longer a synchronization issue, I suggest we close this ticket out. Thoughts?

Steve