New API and codegen for Replicated Entities

[ennru]

We're reviewing and simplifying the APIs for replicated entities https://github.com/lightbend/akkaserverless-framework/issues/698 to make them available for users during the open beta phase.

How can we support using Replicated Entities with codegen?

[ennru]

See even https://github.com/lightbend/akkaserverless-java-sdk/pull/26

[patriknw]

This shouldn't be started until we have finalized the SDK APIs. There are several changes to be completed first. https://github.com/lightbend/akkaserverless-framework/issues/698#issuecomment-858541797

[pvlugter]

This will be switched to the new approach first, without annotations or reflection.

This will also need codegen annotations added to the protocol in framework, including a way to specify the replicated data types (with key/value types where appropriate).

[patriknw]

including a way to specify the replicated data types (with key/value types where appropriate)

that might become rather verbose? is there a way to require that is specified in the concrete ReplicatedEntity class instead? the downside of that suggestion might be that the first generated class will not compile when those types are unknown

[pvlugter]

including a way to specify the replicated data types (with key/value types where appropriate)

that might become rather verbose? is there a way to require that is specified in the concrete ReplicatedEntity class instead? the downside of that suggestion might be that the first generated class will not compile when those types are unknown

Yeah, the alternative would be generate something without the types. Could be confusing either way — specifying the extra types, or having something that only partially generates.

So I was expecting the code to have the replicated data type: so that it's ReplicatedEntity<ReplicatedCounter> or for shopping cart map just as a test example might be ReplicatedEntity<ReplicatedRegisterMap<String, ShoppingCartDomain.LineItem>>.

So where the codegen annotations have this for Event Sourced Entity:

option (akkaserverless.file).event_sourced_entity = {
  name: "ShoppingCart"
  entity_type: "eventsourced-shopping-cart"
  state: "Cart"
  events: ["ItemAdded", "ItemRemoved"]
};

That could maybe be something like this for a Replicated Entity:

option (akkaserverless.file).replicated_entity = {
  name: "ShoppingCart"
  entity_type: "replicated-shopping-cart"
  replicated_register_map: {
    key: "String"
    value: "LineItem"
  }
};

[patriknw]

That looks pretty good. I agree that it's good to refer to the proto value (LineItem) like that, since that is similar to how events and state are defined.

[pvlugter]

Not sure what the replicated data types that don't need any extra settings would be: replicated_counter: {} or replicated_counter: "" or something is a bit awkward. Could also do something where there are optional key and value for when these are used, with an enum for the data type. Enums are required by convention to have prefix though. Then it would be:

option (akkaserverless.file).replicated_entity = {
  name: "ShoppingCart"
  entity_type: "replicated-shopping-cart"
  data_type: REPLICATED_DATA_TYPE_COUNTER
};

option (akkaserverless.file).replicated_entity = {
  name: "ShoppingCart"
  entity_type: "replicated-shopping-cart"
  data_type: REPLICATED_DATA_TYPE_REGISTER_MAP
  data_key_type: "String"
  data_value_type: "LineItem"
};

Or for a map that only needs key type:

option (akkaserverless.file).replicated_entity = {
  name: "ShoppingCart"
  entity_type: "replicated-shopping-cart"
  data_type: REPLICATED_DATA_TYPE_COUNTER_MAP
  data_key_type: "MyMessageType"
};

Or for a set with element/value type:

option (akkaserverless.file).replicated_entity = {
  name: "ShoppingCart"
  entity_type: "replicated-shopping-cart"
  data_type: REPLICATED_DATA_TYPE_SET
  data_value_type: "MyMessageType"
};

[patriknw]

Can we get get better protoc validation from the replicated_register_map: { approach?

I think replicated_counter: {} can work. At least it is consistent for all types and makes it easy for us to evolve/add more things to it if needed.

[pvlugter]

Continuing the discussion on switching away from the mutable API for replicated data in https://github.com/lightbend/akkaserverless-java-sdk/pull/270#discussion_r701698989

We can have specialised replicated entity classes, with their own specific effects.

class MyCounter extends AbstractCounter { // extends ReplicatedCounterEntity
  // optional empty value could be in override method — for something other than the 0 default

  @Override public Effect<Response> someMethod(long currentValue, Request request) {
    return effects()
        .increment(request.getValue()) // specific effects for each replicated data type
        .thenReply(Response.newBuilder...);
  }
}

Because the Java Set and Map interfaces have a bunch of mutable methods, that would have to throw unsupported operation exceptions, rather than passing the current values using those interfaces we may want to keep the Replicated* interfaces but with only the read methods. This would be clearer for sets and maps, and it would probably be clearest to do this for all types, so the current value for a counter is not the long value directly, but a ReplicatedCounter with get still.

class MyCounter extends AbstractCounter { // extends ReplicatedCounterEntity
  @Override public Effect<Response> someMethod(ReplicatedCounter counter, Request request) {
    // counter.get() to read current value, no mutating methods
    return effects()
        .increment(request.getValue()) // specific effects for each replicated data type
        .thenReply(Response.newBuilder...);
  }
}

We still have type parameters for keys or values where required:

class MySet extends AbstractSet { // extends ReplicatedSetEntity<MyElement>
  @Override public Effect<Response> someMethod(ReplicatedSet<MyElement> set, Request request) {
    // set.contains(element) and so on, for read-only methods
    return effects()
        .add(request.getSomething()) // specific effects for each replicated data type
        .remove(request.getAnother()) // multiple primary effects is probably simplest
        .thenReply(Response.newBuilder...);
  }
}

class MyCounterMap extends AbstractCounterMap { // extends ReplicatedCounterMapEntity<MyKey>
  @Override public Effect<Response> someMethod(ReplicatedCounterMap<MyKey> counterMap, Request request) {
    // counterMap.contains(key), counterMap.get(key) and so on, for read-only methods
    return effects()
        // counter-specific effect builders
        .update(request.getSomeKey(), effects -> effects.increment(request.getSomeValue()))
        .thenReply(Response.newBuilder...);
  }
}

The ReplicatedMap can store any other replicated data type, and currently has a getOrCreate method accepting a replicated data factory. This doesn't work with the switch to specialised replicated entity classes, and the read-only interfaces. As we now have the specialised replicated maps (ReplicatedCounterMap, ReplicatedRegisterMap, ReplicatedMultiMap), the motivation for using the generic replicated map will be to have a heterogeneous map that combines multiple replicated data objects of different types. We could remove ReplicatedMap altogether, but combining CRDTs still seems useful to support. This could be changed to more of an HMap interface with typed keys:

class MyMap extends AbstractMap { // extends ReplicatedMapEntity<MyKey>

  // the keys can have specialised types, like entities, for have specific effects more easily
  // the key type can still be anything akka serverless serializable, like protobuf message or primitives
  private ReplicatedMap.CounterKey<MyKey> counter1 = ReplicatedMap.createCounterKey(MyKey.for("counter1"));
  private ReplicatedMap.CounterKey<MyKey> counter2 = ReplicatedMap.createCounterKey(MyKey.for("counter2"));
  // the replicated data types that take parameters would accept them on the key
  // note: at this point, it's looking quite similar to creating CRDTs again
  private ReplicatedMap.SetKey<MyKey, MyElement> set1 = ReplicatedMap.createSetKey(MyKey.for("set1"));
  // and what about nesting replicated maps?

  @Override public Effect<Response> someMethod(ReplicatedMap map, Request request) {
    // map.get(counter1) to get the ReplicatedCounter read-only interface for that counter
    // map.get(set1) to get the ReplicatedSet<MyElement> read-only interface
    return effects()
        // specific effects for each replicated data type based on the typed key
        .update(counter1, effects -> effects.increment(request.getSomeValue())
        .update(counter2, effects -> effects.increment(request.getSomeOtherValue())
        .update(set1, effects -> effects.add(request.getSomething()).remove(request.getAnother()))
        .thenReply(Response.newBuilder...);
  }
}

Thinking about ReplicatedMap and the usefulness of combining different CRDTs leads to another idea, which I'll put in its own comment.

[pvlugter]

The automerge project, and the JSON CRDT paper before it, support replicating a JSON-like object with Map and List CRDTs. Looking at the use cases for a heterogeneous ReplicatedMap, a different approach to the above specialised entity types that we could explore is a more automatic conversion of Protobuf messages (or JSON) to a CRDT, similar to automerge.

The default conversion for a Protobuf message could be to a replicated map of registers. While we could replicate the whole object as a register, having each field be its own register would already allow different fields to be updated independently without conflicts.

We would probably need a sequence / list / ordered set CRDT for repeated fields, which we don't have currently.

This would require replicated data specific annotations on the fields for the state Protobuf message. And we'd need to generate the effects API from this, or create a diff between values.

Note that generated effects specific to each field would also start looking like functional lenses, which we may want to start generating for Java for mutating deeper Protobuf structures anyway (ScalaPB already does this too).

This would mean that a replicated entity has a state object, like value entities. I think it already makes sense to see replicated entities as replicated value entities, and that it should start becoming replication across geographic regions or to clients (part of serverless at the edge ideas). Having this approach, where there's a Protobuf message for state with CRDT semantics applied to it, could bring these ideas together more.

So what could this end up looking like: the file-level codegen annotation would be like a value entity:

option (akkaserverless.file).replicated_entity = {
  name: "Something"
  entity_type: "something"
  state: "Thing"
};

While the Thing message needs replication semantics added:

message Thing {
  // defaults to LWW register, if it's not explicitly a counter
  int64 number = 1;
  // to specify a number should have PN counter semantics
  int64 counter = 2 [(akkaserverless.field).replicated_counter];
  // another message becomes a nested map
  AnotherThing another = 3;
}

message AnotherThing {
  // a repeated field could have set semantics, or otherwise default to a sequence CRDT (or to LWW register)
  repeated string tags = 1 [(akkaserverless.field).replicated_set];
  // while a message is already treated as a replicated map, a dynamic map could be created using protobuf map
  // or maybe this is automatic, rather than explicitly annotated
  map<string, int64> dynamic_counters = 2 [(akkaserverless.field).replicated_counter_map];
}

For the generated entity, there could be two approaches. One is to have something that's the same as value entity, and a new state is passed and the current and new states need to be diff'ed to create the replication delta:

class Something extends AbstractSomething { // extends ReplicatedEntity
  @Override public Thing emptyState() {
    return Thing.getDefaultInstance();
  }

  @Override public Effect<Response> someMethod(Thing currentState, Request request) {
    // read from the state as a regular protobuf object
    return effects()
        // We set a new state, and a replication delta is created automatically.
        // For a counter, this compares the current and new states to get an increment delta.
        // While for registers, it just sets to the new value; and so on.
        .updateState(
          currentState.toBuilder()
            .setNumber(request.getValue())
            .setCounter(currentState.getCounter() + request.getIncrement())
            .setAnother(
              currentState.getAnother().toBuilder()
                .setTags(addToList(currentState.getAnother().getTags(), request.getSomething())))
            .build())
        .thenReply(Response.newBuilder...);
  }
}

Or effects are generated for the fields that are typed specifically:

class Something extends AbstractSomething { // extends ReplicatedEntity
  @Override public Thing emptyState() {
    return Thing.getDefaultInstance();
  }

  @Override public Effect<Response> someMethod(Thing currentState, Request request) {
    // read from the state as a regular protobuf object
    return effects()
        .update(
          // effect builder generated specifically for fields in Thing, looks similar to lenses
          thing -> thing.number.set(request.getValue()),
          thing -> thing.counter.increment(request.getIncrement()),
          thing -> thing.another.tags.add(request.getSomething()))
        .thenReply(Response.newBuilder...);
  }
}

For something like just a replicated counter, it would still be wrapped in a message, just like a value entity counter. The conversion could optimise this and avoid the replicated map wrapper though.

Anyway, not sure yet if there are any barriers to doing this, or whether we don't want to tie this so much to Protobuf, but I think it's worthwhile considering this API for replicated entities, which is much more of a replicated value entity.

[patriknw]

I think my preference would be to not have generated effects, something like this:

class MySet extends AbstractSet { // extends ReplicatedSetEntity<MyElement>
  @Override public Effect<Response> someMethod(ReplicatedSet<MyElement> set, Request request) {
    // set.contains(element) and so on, for read-only methods
    ReplicatedSet<MyElement> updatedSet =
      set
        .add(request.getSomething())
        .remove(request.getAnother());
    return effects()
        .update(updatedSet)
        .thenReply(Response.newBuilder...);
  }
}

Looking at that I wonder how important it is to have them immutable? The strange thing in my opinion would be to not have the explicit effects().update(updatedSet). I'm fine with changing them to immutable, but it might not be critical for the Java dev experience.

The second idea of Thing currentState is interesting, but too big change right now, since we want to wrap up this week. It is definitely something we should create a separate issue for and evaluate more.

[pvlugter]

Ok, so that's the immutable + update effect approach that was discussed earlier.

my preference would be to not have generated effects

Effects wouldn't be generated, but specific to each replicated data type (each specialised replicated entity class). So a ReplicatedCounterEntity would have increment and decrement effects, while a ReplicatedSetEntity would have add and remove effects, and so on.

Looking at that I wonder how important it is to have them immutable? The strange thing in my opinion would be to not have the explicit effects().update(updatedSet). I'm fine with changing them to immutable, but it might not be critical for the Java dev experience.

The only thing with keeping it mutable will be what happens if it's not passed to update? The underlying structure has been changed. Do we end up with a "clone" first approach, to make sure that mutations without update effect are discarded?

Also, creating an updatedSet would need to be a new immutable API. That's not possible now as the current add / remove (implementing java.util.Set) return boolean, given that it's the mutable API. So leaving the current mutable APIs + requiring an update effect would need to be:

class MySet extends AbstractSet { // extends ReplicatedSetEntity<MyElement>
  @Override public Effect<Response> someMethod(ReplicatedSet<MyElement> set, Request request) {
      // copy of replicated set is actually passed, to ensure it's passed to update or otherwise ignored
      set.add(request.getSomething());
      set.remove(request.getAnother());
    return effects()
        .update(set)
        .thenReply(Response.newBuilder...);
  }
}

[patriknw]

Effects wouldn't be generated, but specific to each replicated data type (each specialised replicated entity class). So a ReplicatedCounterEntity would have increment and decrement effects, while a ReplicatedSetEntity would have add and remove effects, and so on.

Got it!

I still like the consistency of only one primary effect updateState in the same way as with value entity.

Do we end up with a "clone" first approach, to make sure that mutations without update effect are discarded?

I think that is fine. If it's a performance problem we could make a shallow clone and only deep clone on the first mutating operation.

[pvlugter]

Follow up issues: #298 (immutable data structures) and #299 ("auto replication" idea).