asynchronics / asynchronix

High-performance asynchronous computation framework for system simulation
Apache License 2.0
170 stars 9 forks source link

Ensuring determinism with periodics #34

Open joshburkart opened 3 weeks ago

joshburkart commented 3 weeks ago

I'm wondering about determinism in the following simplified situation: Say we have two models, A and B, each with an execute method that we schedule to run periodically at the same periodicity and no phase offset. Thus for each cycle, A::execute and B::execute will run at the same simulated time. Say further that the execute method for A immediately sends an event to B, which modifies B's state, affecting what B's execute method will do. As a result, without a way of specifying which periodic executes first each cycle, there appears to be a race condition possible where the outcome of the simulation isn't fully determined/predictable.

Am I understanding this right? Are there ordering guarantees for periodics? Is there a way to ensure determinism, other than something hacky like adding tiny phase offsets? Would appreciate any context that can be provided. Thanks so much!

sbarral commented 3 weeks ago

A subtle topic indeed!

TL;DR: it is not possible to ensure determinism with schedule_periodic_event, but if you are willing to live on the bleeding edge this is possible with the development branch (main) using the schedule method and periodic Actions.

In general, the scheduler attempts to strike a balance between efficiency and determinism. The minimum guarantee when it comes to determinism is that the execution order must be preserved within messages addressed to the same model. To achieve this, when events are polled from the scheduler queue (which preserves scheduling order), all events targeting the same model are fused into a single sequential Future and this fused Future is spawned onto the executor. However, events targeting different models will end up in different Futures so even though they are spawned in order, there is no telling in which relative order the multi-threaded executor will run them.

In the development branch, EventSources were introduced which are basically free-standing Outputs that can schedule Actions that broadcast the same message (possibly periodically) to many models. But since several Actions scheduled for the same timestamp may have one or mode target models in common, in order to preserve the execution order within messages to the same model, the scheduler needs to be conservative and fuse all Actions in a single sequential future.

You can take advantage of that by scheduling two periodic Actions using two EventSource s, one targetting only model A and one targetting only model B, relying on the fact that the scheduler will fuse the Actions into a sequential future. While this is not documented behavior, I don't see that changing in the future. Note that you cannot use a single EventSource that broadcasts to A and B because for efficiency reasons, all broadcast operations resolve futures in parallel so ordering is no guaranteed (similarly to futures::stream::FuturesOrdered).

The scope for version 0.3 keeps on expanding so I certainly won't consider this for the next version, but some simulators have a concept of super-dense time where the timestamp has an additional integer part that make it possible to decompose a time step into several sub-steps. AFAIK this concept does not exist in the ECSS SMP standard which suggests it might be a bit niche, but if you feel the need for that at some point, feel free to open a specific issue.

sbarral commented 3 days ago

This issue prompted further internal discussions and we came to the conclusion that the current ordering guaranties for scheduling are not fully consistent with the ordering guaranties for non-scheduled messages (those sent from Output or Requestor ports).

What we should have done in retrospect is to guaranty ordered sequential execution for same-time events based on the model that scheduled them rather than based on the models targetted by such events. Doing this would provide exactly the same kind of determinism as for regular messages: if model A schedules message M1 to model B and then schedules message M2 to model C which upon receipt of M2 sends message M3 to model B, it is guaranteed that B will process M1 before M3.

In this context it makes sense to treat the main scheduler as a model, which would give exactly the determinism you expected, whether an event or an action is used.

This is thus added to the v0.3 roadmap, but it shouldn't be too difficult to implement this behavior in the 0.2 branch and it is very unlikely to break existing user code. Let me know if you need it now, we could try to make a point release.

joshburkart commented 3 days ago

That makes sense -- thanks so much! I don't need it urgently but definitely a desirable feature in the next 3 months or so for me.

joshburkart commented 2 hours ago

I had a follow-up question: after the changes you described are implemented, under what conditions will an Asynchronix simulation be fully deterministic? Will it be sufficient for all constituent models to run deterministically and possess no shared state?