search

Podcastindex-org / podping-hivewriter

The hive writer component of podping.
MIT License
15 stars 5 forks source link

Implement new request/response formats in ZeroMQ listener #16

Closed agates closed 2 years ago

agates commented 3 years ago

The current 1.0 version only listens for straight string-based IRIs -- one per message.

We need to add a way to receive more information than just the IRI, including (from most to least important):

  1. "reasons", which indicate why the IRI is being sent
  2. MIME types, which indicate the type of the content associated with the IRI
  3. Request for health/status

To retain backwards compatibility we should first check if a request is a valid IRI, and otherwise assume it is the new message schema.

This should also include a new response format depending on the request type. Currently we send either "OK" or "Invalid IRI". The new response should be a formalized response format indicating success/error/health/status.

I am leaning toward either JSON or msgpack for the new formats -- probably JSON for compatibility, but could be convinced otherwise.

agates commented 2 years ago

Since we decided we're going to integrate more closely with podping.cloud, I'm going to opt for a binary format with a standardized schema for this. Having used Cap'n Proto in the past (in python with capnpy), and seeing it has good rust bindings as well, that's what I'm going to start with.

Wondering about whether or not to implement the MIME type in this, as it seriously complicates matters for the resulting hive payload. Maybe consider adding it to the operation_id as discussed in #18? Not sure if it's worth it.

agates commented 2 years ago

1.1 will not yet change the ZeroMQ listener in order to get a minimum viable product for "mediums" and "reasons" via CLI options.

Moving to version 1.2.

daveajones commented 2 years ago

Do you have an idea of how the front-end should pass those values to hivewriter over the zmq channel?

agates commented 2 years ago

Do you have an idea of how the front-end should pass those values to hivewriter over the zmq channel?

Current plan is a capnproto schema that we can generate both python/rust code from. It encodes to binary seamlessly. Open to other ideas but am particularly interested in having a schema as the source of truth.

Rust library I have in mind: https://github.com/capnproto/capnproto-rust

EDIT: To be clear I don't have any plans to use the RPC component at the moment.