ckanext-validation and table designer synergies 💃🕺

[wardi]

Following dev meeting discussion: ckanext-validation and the upcoming table designer feature both define resource data schemas and validation rules.

Let's find commonalities and adopt a plan to make these systems work together as well as possible.

Comparing ckanext-validation and table designer

	ckanext-validation	table designer
data source	csv/excel file	datastore table
data schema	Table Schema json in resource	datastore data dictionary
validation	Table Schema Python library sync or bg job	datastore triggers/python error formatting
datastore API	read only	read/write

Ideas for integration

• allow easy conversion between ckanext-validation and table designer resources
  • mapping types and validation rules (best-effort)
  • one click to convert in either direction or create a new resource based on an existing one
• create common schema detection and generation tool like datapusher+/qsv for existing files @jqnatividad
  • able to import as either table designer or ckanext-validation resource with auto-generated schemas
  • reliably detect controlled lists and data types
• use common UI for customizing schemas
  • an inline ckanext-validation schema editor could use some of the same snippets/UI for defining table schema json
• generate datastore API documentation from actual data and either type of data schema
  • provide copy+paste examples that actually work for the specific resource

Other ideas?

[amercader]

Thanks for putting this together @wardi . Let me prepare a quick demo as well as write up our long term vision for this so it's easier to discuss. I think there is a lot of overlap between the two features and even with some of the points raised by @jqnatividad datapusher+ discussions to have a wider discussion around data ingestion, description and validation. I'll try to put something together by next week (it's a holiday week here)

[wardi]

That sounds really interesting @amercader I'm looking forward to it.