Subkernels' RTIO timestamp does not match master kernel

[Spaqin]

Subkernels RTIO timestamp is separate, thus calling a subkernel and immediately sending an RTIO event will usually cause an underflow.

Minimum example:

    @kernel
    def run(self):
        self.core.reset()
        self.ts()
        print(now_mu())
        print(rtio_get_counter())  # similar (I assume they should be)
        subkernel_await(self.ts)

    @subkernel(destination=1)
    def ts(self) -> TNone:
        print(now_mu())
        print(rtio_get_counter())  # very different
        delay(300*ms)  # to account for subkernel calling overhead
        self.ttl.pulse(100*ms)  # underflow

Workaround within the subkernel that would synchronize the RTIO timestamps (mostly?):

         at_mu(rtio_get_counter())

or just self.core.reset(), and keep the timelines separate.

Should subkernels share the RTIO timestamp? At least from the moment of their call? Or should drtio It could be manipulated with delays, at_mus, etc. so it seems a valiant effort to keep it perfectly synchronized. And TSC is synchronized from master, so if a subkernel does timeline manipulation and calls another subkernel, it would make no effect.

So it could be either an automated synchronization, or it should be solved with an extra note in the documentation.

[sbourdeauducq]

Should subkernels share the RTIO timestamp? At least from the moment of their call?

Yes it makes sense to me that the value of now_mu at the beginning of a subkernel should be the value of now_mu from the caller at the time of the call. Then it should of course be a variable local to the device on which the subkernel is executing, otherwise subkernels running concurrently would interfere with each other.

This sounds like a simple and reasonable thing to do and not a "valiant effort", or am I missing out something?

And yes DRTIO already synchronizes the TSC i.e. the values returned by rtio_get_counter().

[Spaqin]

or am I missing out something?

I was already thinking about having it synchronized, to match the value. But passing the current now_mu on subkernel call would be a reasonable expectation - the called subkernel will be a little behind due to latency, but within the ballpark; that's not too bad.

It will be also necessary to clear up some things in the manual later too, e.g. that rtio_get_counter will return master's time, rather than local.

[sbourdeauducq]

The DRTIO latency impacts the constant difference between the TSCs on different devices, so the same timestamp on different devices will produce events which are offset by a constant time. But this is not different from e.g. having different cable lengths at the outputs of the same DDS card.

rtio_get_counter() shall return local time, otherwise it will be slow and also make it more difficult to deal with underflows.

[dhslichter]

As long as the DRTIO latency is not extremely long (multiple microseconds, such that it's much bigger than typical cable delays), and as long as it is unchanged across power cycles (is this the case?), then it can be calibrated out in the same way that cable lengths would be calibrated out. Nondeterminism in this latency due to power cycling would already be affecting some experiments even in the absence of subkernels, so that's a bigger question. I think that for logical clarity it is fine to have the subkernel and its caller have the same now_mu at the start of the subkernel as discussed. This should of course be noted in the documentation, as well as the need to consider the DRTIO latency and calibrate for it if the application is sensitive to this.