Calling Cove as a Python library (without the web interface)

OpenDataServices / cove

CoVE is an web application to Convert, Validate and Explore data following certain open data standards - including 360Giving, Open Contracting Data Standard, IATI and the Beneficial Ownership Data Standard

http://cove.opendataservices.coop

Other

43 stars 11 forks source link

Calling Cove as a Python library (without the web interface) #207

Closed Bjwebb closed 6 years ago

Bjwebb commented 8 years ago

Some data publishers want to be able to use Cove's additional/custom checks in their tests.

We also have a usecase for making various checks of the sample data https://github.com/open-contracting/sample-data/issues/11 https://github.com/open-contracting/sample-data/issues/7

kindly commented 8 years ago

The particular library functions we should consider are.

Validation Errors (in cove we modify a few of the json schema validation to be more suitable to the schema)
Additional Fields (the report as to extra fields the data has)
Flattentool warnings (possible errors in the conversion between the spreadsheets and the json)
Summary Statistics (aggregations of summary statistics of the data)
Unflattening/Flattening (this could be done in flattentool directly but it would be really useful to have the config options set up for you in the exact way cove does)
Any new feature that is appropriate should have a relatively easy path to becoming a library function.

The one thing that is uncertain is the how much we should reuse the main config file for this library and how we allow changing options within it in the library (i.e what schema we choose) should be managed.

Bjwebb commented 8 years ago

As I understand it @robredpath has done some good refactoring to help with this, but we still need to think about some of the considerations above for providing a useful API. One way to proceed on this is to use the API ourselves outside of Cove - one example would be https://github.com/open-contracting/sample-data/issues/7

http://cove.readthedocs.io/en/307-sphinx-docs/lib/ is possible an overview of what functions exist in cove.lib now.

timgdavies commented 7 years ago

Building on Ben's comment above, we have a use-case coming up again for OCDS sample-data repository, where @annapowellsmith is currently working on collating large corpus of published OCDS data for analysis.

In this process it would be very useful to have the ability to use the full features of the OCDS validator at the command line to:

Detect release or record package, version and extensions;
Validate against the appropriate schema;
Report on extra fields, codelist validation etc.

I can see that release-level validation could end up being very desirable here too (i.e. reporting on which releases fail validation), but I realise that is a separate issue.

robredpath commented 7 years ago

@timgdavies I think that 'CoVE as a library' is a somewhat different proposition from 'CoVE on the cmdline' - there may be plenty of overlap in how we implement them, however.

How much flexibility do we have over the interface? Does this need to be a neat cmdline tool from the off, or is it OK to use Django's manage.py and require the user to be running in a virtualenv to use it?

timgdavies commented 7 years ago

This is a good question: if starting from Django manage.py in a virtualenv is a step on the way to something more, and we can clearly document usage, that sounds sensible to me - but open to other views from the community.

robredpath commented 6 years ago

We've got a few users using the cmdline version of CoVE, and in research I did try to tease out if anyone wanted a library version. No-one did. Closing - we can open a new issue to explore this if/when someone asks!