PubSub

[matt2e]

Design doc is here.

Major steps:

• [x] #1594
• [x] #1599
• [x] https://github.com/TBD54566975/ftl/pull/1608
• [x] #1607
• [x] #1595
• [x] #1596
• [x] #1598
• [x] #1658
• [ ] #1674 (when?)

Issues:

• [x] #1640
• [x] #1642
• [x] #1660
• [x] #1685
• [x] #1703

[matt2e]

Case to handle:

• Deployment registers a topic with event A
• Updated deployment happens, now with the topic defined with event B
• ... what should happen?

[matt2e]

Case to handle:

• Deployment registers a subscription
  • Some events are consumed by the subscription. Curser points to event row A
• Replace deployment, now without the subscription
  • Quick thoughts: Does that subscription row just sit there untouched? Should we be deleting it? Will this work ok when there are multiple versions of a module? Don't worry about that for now?
• ... Wait a week
• Replace deployment again, now with a subscription with the same name as the old subscription
• ... is the cursor still pointing to a week ago?

[matt2e]

Thing to think about: When a module is updated and removes a topic, should the topic just sit there?

• The module won't be able to publish any new events anyway...
• It allows time for subscribers to catch up to the remaining unhandled events, which is good

[matt2e]

Case to handle:

• [x] Can not publish to external module's topic

[matt2e]

Queries are doing a lot of joins. Should we denormalize module name into topics?

[alecthomas]

For now we won't worry about catching any of these cases, we want a holistic approach to managing schema changes across FTL.

[alecthomas]

Queries are doing a lot of joins. Should we denormalize module name into topics?

Yeah, we do it in other places, so if you think it's necessary go for it.

[alecthomas]

I'm going to close this out. We can revisit later if we decide to do batching. Nice work Matt.