Rethink where to generate absolute timestamps (service endpoint vs. runtime)

[tillrohrmann]

We should rethink where we generate absolute timestamps for time based operations (e.g. sleeps, delayed calls). At the moment, the SDK generates a journal entry with an absolute timestamp when the operation should be executed. An alternative could be to store the relative duration and let the runtime convert it into an absolute timestamp when receiving the command from the service endpoint.

Generating the absolute timestamp on the runtime

Pros:

• Being less affected by clock skews from the machine on which the services are running
  • Within each Raft group, the peers could agree on the current time (guaranteeing monotonically increasing time)

Cons:

• Buffering of commands (e.g. on the service endpoint) will cause an additional delay for time based operations

[slinkydeveloper]

In the context of this decision we should also include/reason about:

• https://github.com/restatedev/sdk-java/issues/16. If we claim to have a lower bound after which we cannot provide a meaningful timer, then being extremely precise might not be necessary?
• Perhaps we should investigate what is the order of magnitude of clock skew in cloud deployments. Some pointers:
  • https://aws.amazon.com/blogs/mt/manage-amazon-ec2-instance-clock-accuracy-using-amazon-time-sync-service-and-amazon-cloudwatch-part-1/
  • https://learn.microsoft.com/en-us/azure/virtual-machines/linux/time-sync
  • https://cloud.google.com/compute/docs/instances/configure-ntp