search

NOAA-GSL / VxIngest

Other
2 stars 0 forks source link

Add any changes needed to data model for federated database #56

Closed venitahagerty closed 3 years ago

venitahagerty commented 3 years ago

Currently, most documents in the model have a version. Does the data model need any changes for the federated database, particularly related to version? Will we have mixed versions in the same bucket? Will we require all federated users to upgrade their data and apps to stay in the federation? Can apps read multiple versions of documents? These questions need to be discussed and resolved sometime after the "steel thread" is completed.

randytpierce commented 3 years ago

It seems to be sufficient for the moment, we can close this and recreate an issue if there is a problem in the future.

randytpierce commented 3 years ago

Personally I think documents should always be versioned. They should also have a specific Time To Live so that they can expire rationaly. Fortunately TTL is automatically part of the couchbase metadata, although for the moment we always set it to essentially infinite. It might make sense to always have a data creation date as well but we haven't fully developed the code to support that part of our model yet.

I don't foresee an app using different versions of the same data simultaneously but an app certainly might use a specific version of a specific set of data and another instance of the app might use a different version. An app might also use a specific version of the metadata while using a different version of sums data, for example.

It can be very useful when you change a data ingest routine, for example. It allows the introduction of data without overwriting existing data, allowing an opportunity for checking out the new data while the current data marches on. Versions can easily be changed in place, if need be. I realize that this can be accomplished with scopes, collections, and using different buckets, but if only some of the data changes you can end up in a situation where the app has to deal with different scopes, collections, or buckets simultaneously in order to get to data that is really just different versions of similar data. I mean, for example, most of the metadata could be the same but just part of it needs a new version temporarily. The app can access the temporary data with a version while accessing the remaining data the same. Without versions you might end up needing to duplicate the entire set of metadata. The version of a dataset can also be controlled with metadata, allowing an app to access different datasets without the need to redeploy the app.

Is there a problem with versioning the data that we need to solve? Is there something pertaining to a federated database that is incompatible with versioned data? I don't foresee any myself. randy

On Tue, Aug 31, 2021 at 10:58 PM Venita Hagerty @.***> wrote:

Currently, most documents in the model have a version. Does the data model need any changes for the federated database, particularly related to version? Will we have mixed versions in the same bucket? Will we require all federated users to upgrade their data and apps to stay in the federation? Can apps read multiple versions of documents? These questions need to be discussed and resolved sometime after the "steel thread" is completed.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/NOAA-GSL/VxIngest/issues/56, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGDVQPXQ7JXGF6PFA5ZWYLTT7WXHFANCNFSM5DFSYIKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Randy Pierce