Open pdelboca opened 3 years ago
:+1: I like the idea of fault tolerance and graceful degradation. I think migrations are much easier if systems allow for eventual consistency rather than expect to be consistent 100% of the time.
My thought initially was to ensure that if a dataset doesn't exist in the metastore we rely on CKAN data and "migrate" on demand either when the dataset is saved for the first time, or when it is read for the first time (but I think this is slightly less preferred).
I am not sure how to prioritize this, but I will look into the complexity of this.
Nowadays there is no handling for possible discrepancies in the two backend we are using to store metadata. This causes CKAN to display a dataset (because it exist in Metastore) but fail when trying to edit it because it doesn't exist in
metastore-lib
backend (for example, data has not been migrated into github repositories).Example traceback:
This will also introduce a hard dependencies: we cannot change/update the
metastore-lib
backend without a data migration which is something that shouldn't happen but it is worth to have it in mind while we are in the development workflow.Is this gonna be handle in a specific way?
Some scenarios I can think: