AT: Harvesting problems / download links recognition

[manilly]

Hi, we (AT) triggered a harvesting on Thursday wich seems to be stuck. Could you please have a look into this? Thanks,, Manuel

[jescriu]

Dear @manilly, We restarted the INSPIRE Geoportal services and manually executed the harvesting from the AT endpoint, which seem to be successful. Please, check and let us know in order to close this issue.

[oberseri]

Dear @jescriu

thanks for manually starting the harvesting. It ran through, but we could not publish the results, because we lost over 200 "downloadable datasets". There have been no changes since the last succesful harvesting (in most cases, I could not check them all). I just tested 2 examples (50A04BCD-858A-4656-B136-F8DE18D8F9C0 and d583b754-ce4e-4e4f-bb86-0ecd4b10bd43) with the linkage checker an they are completely OK there, but marked as "not downloadable" (no service/no link) in the preliminary harvesting results. In the meantime we have run a new harvesting with more or less the same bad results.

Sorry for this very "unspecific" error report, but I think, there are some "bigger problems" behind.

Best regards

[jescriu]

Dear @manilly, We will have a look and let you know.

[laers]

Dear @jescriu

Can you also have a look at 64?

Perhaps the problems have som overlap. We have the exact same issue.

[manilly]

harvest from Thursday again wasn't successful...

[oberseri]

Dear @jescriu

we are stuck again since thursday 18. (for one week!) - please help!

Best regards

[Privatmijoha]

Dear all. Sweden has lost a lot of download services that we know is good through tests in validator and they have passed test to INSPIRE Geoportal for more than a year. November 19 something went wrong in the harvest and they did not pass the test to INSPIRE Geoportal any longer.

[jescriu]

Dear @manilly and @oberseri, The harvest got stuck again. We re-started again the service and re-run the AT harvest. Hope to have it finished later today. @oberseri - Regarding the "bigger issues" you mentioned earlier, we are just investigating and trying to resolve an issue from BE which may have the same or similar cause. Just to let you know that we are working on it.

[jescriu]

Dear @laers, Will try to analyse the common denominator with #64 - As you mentioned, some of the issues opened in the repository may have a common cause. We are in the process of tracing it.

[jescriu]

Dear @Privatmijoha, Please open a new dedicated issue for Sweden, to better keep trace of it.

[Privatmijoha]

Started a new harvest november 26 10:08. I hope it works better than the one run november 19.

[oberseri]

Dear @jescriu

we just finished another harvesting - situation is unchanged, so we will not publish it. I attached a list of datasets, which were OK at our last succesful harvesting at Nov. 04., but marked as not downloadable now. I testeted a small subset of them with the linkage checker (marked yellow) an they are OK.

What to do now? Start a new harvesting or wait for "new instructions"?

Best regards LostData.zip

@manilly

[jescriu]

Dear @oberseri, Thank you for this list of lost datasets. Our IT Geoportal team is still working in solving the related issue, so I would recommend you to wait till we have progressed further on it. We will contact you through this issue thread in due time.

[oberseri]

Dear @jescriu, just to inform you: The harvesting process has been successfully run through today and we will publish the results.

Best regards

@manilly

[jescriu]

Dear @manilly and @oberseri, As you noticed, our Geoportal IT team already implemented a fix for this bug. We were just testing the fix with other endpoints before contacting you on it. I proceed to close this issue. Please feel free to comment / open again if needed. Thank you for your patience.

[oberseri]

Dear @jescriu, just to inform and prepare you: Our current harvesting showed bad results again, so we started an new one. I will inform you about the results on monday.

Best regards @manilly

[jescriu]

Dear @oberseri and @manilly, Thanks for sharing. Waiting for the outcomes.

[manilly]

Dear @jescriu, the harvest from friday showed the same bad result with dropped download links:

Just to be sure i picked a random dataset with no harvest link during harvesting:

I used the linkage checker for this ressource: dataset MD: https://geometadaten.lfrz.at/at.lfrz.discoveryservices/srv/ger/csw?service=CSW&version=2.0.2&request=GetRecordById&outputschema=http://www.isotc211.org/2005/gmd&elementSetName=full&id=e76c1db4-69ee-4252-aa0e-c7a65cf069f9 view MD: https://geometadaten.lfrz.at/at.lfrz.discoveryservices/srv/ger/csw?service=CSW&version=2.0.2&request=GetRecordById&outputschema=http://www.isotc211.org/2005/gmd&elementSetName=full&id=7c968192-7317-4e0c-840c-f4766cf0e7e8 download MD: https://geometadaten.lfrz.at/at.lfrz.discoveryservices/srv/ger/csw?service=CSW&version=2.0.2&request=GetRecordById&outputschema=http://www.isotc211.org/2005/gmd&elementSetName=full&id=4b3d8a60-20a2-4cc8-a343-2ac31bf6ce50

The result was good - no errors found:

https://inspire-geoportal.ec.europa.eu/resources/sandbox/INSPIRE-54845ebc-5656-11ec-9bed-0050563f01ec_20211206-063532/

Could you please check?

Thanks, Manuel

@oberseri

[jescriu]

@manilly and @oberseri, Thank you for your last harvest and the analysis. We will check again when possible and get back to you.

[manilly]

Hi @jescriu , we tried a harvest again. Unfortunately the same non-comprehensible bad result as last week. Please check asap as the 15th of December (harvest deadline) is next week! Thanks, Manuel

[jescriu]

Dear @manilly and @oberseri, Our IT Geoportal team will work on it as soon as possible. The issue is reopened now. We will keep you posted.

[GIM-Jarrik]

Hi, We have the same issues for the Luxembourg harvest. Not only the download link recognition, but also view service and download service recognition seems to be off.

One example:

I ran the linkagechecker: dataset: https://catalog.inspire.geoportail.lu/geonetwork/srv/api/records/f8fe0daa-cf2d-4cd5-af87-91ecf7b820b9/formatters/xml?approved=true view: https://catalog.inspire.geoportail.lu/geonetwork/srv/api/records/5728c801-8217-41e8-bbae-0213662bc934/formatters/xml?approved=true download: https://catalog.inspire.geoportail.lu/geonetwork/srv/api/records/b90a94a4-de72-480d-a3a1-5004d9544e55/formatters/xml?approved=true

Linkage seems completely fine:

The validation is also not consistent. One time a certain dataset is fine and next time it isn't, while nothing has changed in the meantime.

Thank you to take a look at this! Jarrik

[oberseri]

Dear @jescriu, our current harvesting from today shows the same bad results - the situation is unchanged! Best regards

@manilly

[jescriu]

Dear @oberseri, @manilly We have this issue in the pipeline. Thank you in advance for your patience.

[nppozar]

We have the same issue for the Slovenia harvest #78

[manilly]

Dear @jrc-inspire and @jescriu , I've seen, the monitoring 2021 reports are available. Unfortunately the calculated values for NSi2.x are not satisfying for us. The problem described above is still not fixed.

I tried to test some of the metadata in the linkage checker, which are shown as not-downloadable in current harvesting from 17.12. & monitoring report. ALL of them are correct according to linkage checker!

For example: Data MD (VS MD | DS MD) fbc95ac7-545f-478c-9a38-16125be857e2 (7c968192-7317-4e0c-840c-f4766cf0e7e8 | 4b3d8a60-20a2-4cc8-a343-2ac31bf6ce50) 1a9a1ebc-73ad-4faf-8611-73a755d3735c (c12096d3-2098-4759-8e15-604dcc068396 | bda17d44-e4d8-47bb-bc76-8ed8d1728223) 226f7d26-71bf-4fbd-a1f9-c8510aeddd8a (d8b2b03e-44b1-489f-a265-65512b62d619 | b04525b0-ff45-4445-940a-ab48e43ada1c) 4009ac42-dc0c-436d-b061-2371290d914c (cfd59bec-5c3d-4d37-a9f0-01263a6464d9 | 2ddbad45-4624-4ada-9e34-5cdddc98a8d9) c5204acc-dbc1-4dce-882e-ab2eebab6db1 (766e69da-abd7-4520-b758-41b1f5821429 | bc9e4fc5-79ee-495a-afc2-15b819f4bffb) 526dcc06-b6c0-4928-bb9a-689002aff19b (7c968192-7317-4e0c-840c-f4766cf0e7e8 | 4b3d8a60-20a2-4cc8-a343-2ac31bf6ce50) ....

fyi: @oberseri

[oberseri]

Dear all, maybe this validator issue could play al role, as many Atom-feed download-links are "not available": https://github.com/INSPIRE-MIF/helpdesk-validator/issues/709

fyi: @manilly

[jescriu]

Dear @oberseri and @manilly, Thank you for warning on the possible link to the mentioned Validator issue. We will look into this issue as soon as we close other issues we are working on right now.

[jrc-inspire]

The functionality of the INSPIRE Geoportal backend is being migrated to a GeoNetwork-based architecture.

Despite were not planning to address open issues affecting the current INSPIRE Geoportal backend because of the mentioned reason, we are investigating the possible cause of the reported drop in the number of downloadable datasets for your endpoint. At the moment, without any results.

It would be helpful if you could check in parallel the logs in your infrastructure to identify any potential issues from your side.

Please contact us at JRC-INSPIRE-SUPPORT@ec.europa.eu in case you have any urgent request.

[oberseri]

Dear all, we understand, that you keep you eyes on the future, but please don't forget our bad 2021 monitoring results.

[jescriu]

Dear @manilly and @oberseri,

Our INSPIRE Geoportal IT team made (quite recently) some quality checks and improvements in the deployment of the INSPIRE Geoportal which is available online to all Member States and EFTA countries through the harvest console.

A new harvest of the AT discovery service endpoint was executed on 12th May:

Since you already published the results in the INSPIRE Geoportal through the harvest console, I hope you are now better satisfied with them.

We were just willing to contact you to get feedback on the new results. Please come back to us if needed.

All the best.