search

HIRO-MicroDataCenters-BV / rhio

Peer-to-peer NATS message routing and S3 object sync solution
MIT License
0 stars 0 forks source link

Do not require NATS subjects or S3 buckets to be scoped by public key #75

Closed adzialocha closed 1 week ago

adzialocha commented 1 week ago

This PR removes the requirement for NATS subject or S3 buckets to be scoped by public key outside of rhio, this made the usage of rhio slightly cumbersome. The public key is still a requirement for subscribe configurations, but only to verify the source of the data coming in.

The config has now the following format:

publish:
  s3_buckets:
    - "bucket-1"
    - "bucket-2"
  nats_subjects:
    - subject: "workload.berlin.energy"
      stream: "workload"
    - subject: "workload.rotterdam.energy"
      stream: "workload"

subscribe:
  s3_buckets:
    - bucket: "bucket-1"
      public_key: "<public-key>"
    - bucket: "bucket-2"
      public_key: "<public-key>"
  nats_subjects:
    - subject: "workload.*.energy"
      stream: "workload"
      public_key: "<public-key>"
    - subject: "data.thehague.meta"
      stream: "data"
      public_key: "<public-key>"

To prevent NATS messages from being re-published via gossip after ingestion which is likely to happen when the subjects can collide now (without the public key namespacing), a custom X-From-Rhio header is inserted into each NATS message. With this "marker" we prevent messages to get re-published when they arrive again on a NATS consumer.