The currently pub-sub specification uses websockets. While websockets are great for supporting real-time frontends, they are not the best for backend service event notifications, as they require every subscribing service to keep a websocket connection open to the pod host at all times. Among other things, this makes it impossible to use with common serverless architectures like Vercel or AWS Lambda, both of which have a maximum execution time for functions.
A more flexible and scalable backend pubsub implementation could use a webhook based approach, especially since most of the communication is one way (the pod host notifying the backend services of any changes). At a basic level, it would work very similarly to the pubsub websocket subscription. Any app that has read permissions on a particular file or directory would be able to subscribe (i.e. submit a webhook url) to be notified if that file or files in that directory have changed. One nice thing about webhooks is that the service receiving the webhooks does not need to authenticate itself, as the appropriate permissions would be validated on the pod host side before sending a given webhook. This is a widely used (Slack, Github) and highly scalable pattern for implementing platform integrations, and one that would ideally be supported at the pod host level.
I'm not sure whether this was already proposed as part of the spec process or not, if so I would love to hear what the barriers to acceptance where. If it hasn't yet been proposed, I would be happy to write a more complete spec for how I think a webhook system could work. I just wanted to gauge support for the general idea before wading into the work too deeply.
The currently pub-sub specification uses websockets. While websockets are great for supporting real-time frontends, they are not the best for backend service event notifications, as they require every subscribing service to keep a websocket connection open to the pod host at all times. Among other things, this makes it impossible to use with common serverless architectures like Vercel or AWS Lambda, both of which have a maximum execution time for functions.
A more flexible and scalable backend pubsub implementation could use a webhook based approach, especially since most of the communication is one way (the pod host notifying the backend services of any changes). At a basic level, it would work very similarly to the pubsub websocket subscription. Any app that has read permissions on a particular file or directory would be able to subscribe (i.e. submit a webhook url) to be notified if that file or files in that directory have changed. One nice thing about webhooks is that the service receiving the webhooks does not need to authenticate itself, as the appropriate permissions would be validated on the pod host side before sending a given webhook. This is a widely used (Slack, Github) and highly scalable pattern for implementing platform integrations, and one that would ideally be supported at the pod host level.
I'm not sure whether this was already proposed as part of the spec process or not, if so I would love to hear what the barriers to acceptance where. If it hasn't yet been proposed, I would be happy to write a more complete spec for how I think a webhook system could work. I just wanted to gauge support for the general idea before wading into the work too deeply.