search

ckan / ckanext-dcat

CKAN ♥ DCAT
https://docs.ckan.org/projects/ckanext-dcat
165 stars 145 forks source link

Handle import stage errors #149

Closed kentsanggds closed 5 years ago

kentsanggds commented 5 years ago

What

Our fetch process had stopped because of Validation Errors which were not properly handled in the import_stage this would eventually lead to the process dying after a number of retries. The end user would be unaware of any problems with the data as the status of the job is not updated.

How to review

2 tests have been added to demonstrate the new behaviour which fixes the issue:

test_harvest_does_not_create_with_invalid_tags - demonstrates that it will not create the dataset with invalid data.

test_import_invalid_tags - checks that a Validation Error is handled and would be stored in an error log which the end user will have visibility of.

kentsanggds commented 5 years ago

@amercader any thoughts on this PR?

kentsanggds commented 5 years ago

@metaodi any chance that you can have a look at this PR?

metaodi commented 5 years ago

Looks good to me.

As for your suggestion to catch these errors already in ckanext-harvest: I agree, this would be a good idea. I'll created an issue to track this: https://github.com/ckan/ckanext-harvest/issues/362