search

oam-dev / spec

Open Application Model (OAM).
https://oam.dev
Other
3.04k stars 246 forks source link

Proposal: Data Placement Policy #402

Open chilung opened 3 years ago

chilung commented 3 years ago

Suppose an edge service in Bay Area serve me well and then I fly to New York, another edge service serve me well too. However, the edge service in New York does not know or have my data in Bay Area edge location. A Cloud service does not have this kind of issue. Does or how OAM addresses this development requirement? It depends on the policy on (1) auto data migration or (2) connect to data nearest to the edge service automatically (data is already there, just need to let the edge service know) according to latency requirements and data amount to be migrated. I believe it can be an important topic to realize edge computing.

We propose a mechanism to define the data placement policy: In Cloud, in Edge Read Only, in Edge Read Write Non-Shareable, and in Edge Read Write Shareable.

  1. For those data defined 'In Cloud', the data will stay in Cloud and will not migrate to the Edge where is near users.
  2. For those data defined 'in Edge Read Only', the data will have an instance in Edge near users. It is possible to have multiple instances for the same data entity spread among multiple edge sites. We need to define the behavior of data migration according to storage usage and data expiration policy.
  3. For those data defined 'in Edge Read Write Non-Shareable', the data will have an instance in Edge near users. It is also possible to have multiple instances for the same data entity spread among multiple edge sites. We need to define the behavior of data migration according to storage usage and data expiration policy. It will be further defined whether these multiple instances should be synchronized back and saved to Cloud or not.
  4. For those data defined 'in Edge Read Write Shareable', although the data also has multiple instances among multiple edge sites, state should be maintained consistence among these instances.

In Open Application Model, we can define it as 'DataPlacementPolicy' scope. User can bind DataPlacementPolicy to different Components in his application. The implement of DataPlacementPolicy will handle the data migration detail based on the Scope kind.

resouer commented 3 years ago

@chilung Based on the discussion on gitter, I belive this is in scope of OAM and a valuable use case. I'd suggest we start from including this ApplicationScope from extend category so we can begin to iterate its spec.

Would you like to draft the first version of spec like HealthScope: https://github.com/oam-dev/spec/blob/master/standard/scopes/health_scope.md

It could be:

apiVersion: extend.oam.dev/v1alpha2
kind: DataPlacementScope
spec:
  ...