Backend: message queue architecture

[Southclaws]

Now the product has grown a bit in terms of functionality, the need for a queue has arisen. There are a few use-cases for this and that list will only grow.

unorganised list of ideas/requirements

• All basic operations should emit an event: account signup, content CRUD (posts, datagraph, links, etc)
• Other components of the system can become somewhat less coupled - such as when a content item is created and the semdex component wants to run the text through AI to index it, this should be an event operation not an explicit function call in each place content can be created/updated
• Certain operations may want to be retried - API calls, image resize, etc?
• Certain operations want to be deferred until later, like image processing
• Do we need to allow subscribes to cancel operations? like e.preventDefault() style

Also, this event system should be usable for:

• Webhooks
• WASM plugin invocations
  • When a WASM plugin is loaded, its manifest is read in order to determine which events the plugin wishes to listen to, then those events are treated the exact same as webhooks: they receive the same payload, react to it and then output a response of some kind

Whatever solution is chosen, there must also be a pure Go embedded version that does not depend on a third party service (like RabbitMQ for example) so we can continue to support barebones installations that run entirely on SQLite/Bleve on a single Fly.io machine.

[Southclaws]

Implemented (naïvely) with Watermill - not CQRS, just pubsub, but that may come later if needed.