[EPIC] centralized data store + data automation

[OriHoch]

introduction

• odata currently has 3000 datasets from 30 organizations
• it makes sense to keep using it as a centralized data store
• however, individual sources may want to have their own CKAN instance, where they can organize and display their data according to their requirements.
• Currently these sources are the local municipalities hosted on datacity and hasadna's instance, however in the future other sources may want to use this, for example, an NGO like The Freedom of Information Movement may want to have their own CKAN instance.
• In addition, some instances will require automatic publishing of data from it's own projects (e.g. data from Budget Key DB will be automatically published and updated on Hasadna's CKAN, which will in turn be published to odata CKAN)

suggested solution

• Develop a central automation system will provide configurable flow of data from various sources to CKAN instances
• Odata will be the central data store, and should be hosted on datacity cluster, to centralize the management of all our instances and ensure high-level of service.

[OriHoch]

@akariv

[OriHoch]

Upload via email is currently the main method of data upload to odata, this feature could be adapted to run as part of the central automation system, allowing to use it for other CKAN instances as well, as part of datacity offering.

[OriHoch]

moving odata to datacity cluster will save resources as it will use the centralized resources

[OriHoch]

automatically synced daily to odata:

• https://maale-adummim.datacity.org.il/ --> https://www.odata.org.il/organization/maale-adummim
• https://opendata.hasadna.org.il/ --> https://www.odata.org.il/organization/the-public-knowledge-workshop

[OriHoch]

opened more specific issues: #92 #93 #94