Broadcast Loops

[Stebalien]

In F3, we've run into some pretty severe broadcast loops. From what we can tell, the time cache is basically useless.

The core issue is that we have NO TTLs and/or expiration on messages. Personally, I'd recommend a simple expiration to all messages that's strictly less than the time cache duration. We can even treat the time cache duration as a network parameter, refusing to propagate messages that have an expiration more than that duration in the future.

[vyzo]

I completely agree.

I think we should have ttls in utc seconds, and the signature should probably include them when present.

I think we need a pubsub spec change for this, i'll be happy to support you in the interest group in specs.

[vyzo]

actually expiration in the envelope.

[vyzo]

hrm, including in the signature maybe backwards incompatible, so it can only be advisory as it may be tampered with.

[Stebalien]

Make it a per-topic option.

[Stebalien]

Specifically, repurpose the seqno?

[vyzo]

I think this would break some systems that rely on it.

[vyzo]

Make it a per-topic option.

Sure.

[Stebalien]

I think this would break some systems that rely on it.

Rely on it being sequential?

[Stebalien]

But yeah, making this a per-topic option would go a long way.

[vyzo]

I think this would break some systems that rely on it.

Rely on it being sequential?

Rather monotonically increasing.

[Stebalien]

Rather monotonically increasing.

unixnanos? unixnanos with a counter in case we try to go backwards?

[vyzo]

Rather monotonically increasing.

unixnanos? unixnanos with a counter in case we try to go backwards?

unixnano init and then sequentially increasing. I don't see anything wrong with allowing an initializer, but we just do unixnano without an initializer for now i think.

[Stebalien]

Well, specifically I was thinking about having a guaranteed monotonically increasing clock. I.e.:

1. Get unix nanos.
2. Look at the last sequence number.
3. If less than or equal to the last sequence number, use the last sequence number plus 1.

We aren't going to broadcast one message per nanosecond, but I like being extra paranoid.

[vyzo]

Yes -- the part that is missing is knowledge of the last seqno basically. The application would need to store it somewhere, load it on restart, and pass it in the constructor.

[Stebalien]

The application would need to store it somewhere, load it on restart, and pass it in the constructor.

Can we not just initialize to the current time on reboot? According to kuba, that's what we already do. To deal with fast reboots, we can insert a very tiny sleep before sending the first message.

[vyzo]

sure.