odata as a data lake - containing all raw data

[OriHoch]

a data lake exposes all the raw, unprocessed, schemaless data (as opposed to data warehouse which exposes processed data with well defined schema)

odata should be this data lake, exposing the metadata via CKAN api and the data via the datastore

This data should also be available on Redash to allow joining data from multiple data sources

[OriHoch]

implementation suggestion:

• using metadata on CKAN packages - define how the data will be ingested and made available
• this allows any user to specify how data should be ingested and made available, e.g. for the raw data lake - only specified packages can be marked for ingestion and the table name can be specified.
• could also allow to specify basic ingestion steps to make the data more accessible

[OriHoch]

could overcome some of the CKAN datastore bugs / ingestion problems