Apply table schemas on datastore import

GetDKAN / dkan

DKAN Open Data Portal

https://dkan.readthedocs.io/en/latest/index.html

GNU General Public License v2.0

372 stars 171 forks source link

Apply table schemas on datastore import #3564

Open dafeder opened 3 years ago

dafeder commented 3 years ago

We want to support, through documentation and probably some helper classes added to the codebase, the application of basic column-level metadata (data type, description) at the moment of an import. For now this will need to be implemented via custom event listener on a per-site basis, but there should be an established pattern to create the listener and provide Drupal with table schema instructions to pass to the DB server based on Frictionless Table Schema field definitions. A helper class could be the best way to pass a single field definition, such as...

{}

... and get back the correct instructions for the Drupal schema API to apply to the table.

Examples coming...

dharizza commented 3 years ago

Created initial proof of concept, still in progress.

QA Steps

[ ] Enable the module
[ ] Create a dataset with a resource, the resource should contain a CSV file. For testing you could use the data specified in the README of the schema_updater module.
[ ] Run dktl drush queue:run datastore_import

Pendant

[ ] Fix table not created.
[ ] Choose naming.
[ ] Define if this will be in core.
[ ] Create method to get the frictionless schema.

dharizza commented 3 years ago

@dafeder I just fixed the issue of the table not being created by adding an extra function to do some mapping between types, in theory frictionless data tables have specific types and they not necessarily match the db types, so for testing I added one extra function that adds some mapping there. So there are two things to do:

Create the mechanism for getting the frictionlessSchema (right now I'm hardcoding it)
Improve the mapping between frictionless types and database types, right now I'm checking if the value we pass to the function is one or the other and we return something that the database understands, so the tables are created correctly (but I only tested with our test data that has integers and floats).