search

ckan / ckanext-validation

CKAN extension for validating Data Packages using Table Schema.
MIT License
28 stars 33 forks source link

IPipeValidation Interface #97

Open JVickery-TBS opened 5 months ago

JVickery-TBS commented 5 months ago

feat(dev): IPipeValidation interface;

Based implementation off of the Archiver plugin's IPipe https://github.com/ckan/ckanext-archiver/blob/master/ckanext/archiver/interfaces.py#L9

This would be very useful to plugins like Xloader and Datapusher to check a resource's validation and continue any processes to get the data into the DataStore.

duttonw commented 5 months ago

Hi @amercader,

Are you able to look over this and see if we can get this merged in and a version cut?

If you are too busy, would you like to provide @ThrawnCA or myself maintaining access to keep this plugin fresh?

This work that @JVickery-TBS is doing is very useful in allowing the xloader/datapusher to postpone its operations until we have a green from the validation sub-system that the file won't corrupt the previously loaded datastore.

kind regards,

@duttonw

wardi commented 4 months ago

@amercader why don't we fork this repo to the ckan github org and give @ThrawnCA and @duttonw maintainer permissions there?

amercader commented 4 months ago

@duttonw @ThrawnCA @JVickery-TBS @wardi

I think it's a great idea, as long as the frictionless folks are happy with it. @pdelboca @roll what do you think?

pdelboca commented 3 months ago

@amercader @duttonw @ThrawnCA @wardi @roll @JVickery-TBS

I think moving it to CKAN github org is a great idea. This extension has a lot of potential, I think it will integrate really well with the latest Table Designer feature of CKAN 2.11.

+1

wardi commented 3 months ago

Sounds great, moving the repo is even better than forking it. I don't have permission to move a repo from frictionlessdata but maybe @pdelboca or @amercader do?

amercader commented 3 months ago

Repo transferred.

@duttonw @ThrawnCA you should have write access to this repo (or have a pending invite to join)

duttonw commented 3 months ago

Repo transferred.

@duttonw @ThrawnCA you should have write access to this repo (or have a pending invite to join)

Thanks, invite accepted.

ThrawnCA commented 3 months ago

ThrawnCA you should have write access to this repo (or have a pending invite to join)

I have write access, yes.

duttonw commented 3 months ago

Hi @JVickery-TBS ,

I've got a pickle of a problem on this now that i've been provided repo rights. The master branch is currently not passing 2.9 tests (unsure when it last worked and it was not stuck on docker container changes).

Am a little sad that 2.10 is not in the matrix, I do see that it was uplifted to 2.10 via this PR https://github.com/ckan/ckanext-validation/pull/66/files so :/

While this is sorted, can you open a PR against https://github.com/qld-gov-au/ckanext-validation and have this interface added, while we sort out how to get these diverged tree's amalgamated (maybe with some feature flags).

JVickery-TBS commented 3 months ago

@duttonw Yeah I started upgrading to 2.10 this summer and also found out that it does not support it yet. I started adding in some super basic support: https://github.com/open-data/ckanext-validation/tree/canada-v2.10

But your fork looks really good. Let me know if you are planning to merge a lot of your fork into here ckan/ckanext-validation, and I will merge yours into our fork too.

Will work on putting this IPipeValidation into your fork though.

ThrawnCA commented 2 months ago

@JVickery-TBS You need to update the Solr test container name from the obsolete "ckan-solr-dev" to just "ckan-solr".

duttonw commented 2 months ago

Hi @JVickery-TBS ,

Before I Merge this, can you also update the README.md with what you have changed.