datopian / aircan

💨🥫 A Data Factory system for running data processing pipelines built on AirFlow and tailored to CKAN. Includes evolution of DataPusher and Xloader for loading data to DataStore.
https://tech.datopian.com/flows/
Other
31 stars 7 forks source link

Better way to send resource fields information #22

Open hannelita opened 4 years ago

hannelita commented 4 years ago

At this time they are sent via array in the DAG params (e.g. "schema_fields_array": "['field1', 'field2']") and everything is treated as text type. What are good alternatives? Questions:

hannelita commented 4 years ago

Waiting for @rufuspollock analysis to ETL pipeline