Subject mapping transforms for Streams

Overview

With nats-server 2.10.0 this feature will be released.

Server PR: https://github.com/nats-io/nats-server/pull/3814 https://github.com/nats-io/nats-server/pull/3827

PR contains a lot of details.

Example Stream config with transforms:

{
  "name": "test2",
  "subjects": [
    "a",
    "b"
  ],
  "retention": "limits",
  "max_consumers": -1,
  "max_msgs_per_subject": -1,
  "max_msgs": -1,
  "max_bytes": -1,
  "max_age": 0,
  "max_msg_size": -1,
  "storage": "file",
  "discard": "old",
  "num_replicas": 1,
  "duplicate_window": 120000000000,
  "subject_transform": {
    "src": "fromtest.>",
    "dest": "input.>"
  },
"sources": [
    {
      "name": "test",
      "subject_transform_dest": "fromtest.>",
      "subject_filter" : ">"
    }
  ],
  "sealed": false,
  "deny_delete": false,
  "deny_purge": false,
  "allow_rollup_hdrs": false,
  "allow_direct": false,
  "mirror_direct": false
}

Besides exposing these new fields in the StreamConfig the client library should also automatically fill-in the subject transformation (subject filter and transform destination) fields in the StreamConfig for the bucket for the defined sources for the bucket, such that the source filter is "$KV.<sourced bucket name>.>" and the destination "$KV.<bucket name>.>".

Example of the StreamConfig for a bucket C that sources from KV buckets A and B:

{
  "name": "KV_C",
  "subjects": [
    "$KV.C.>"
  ],
  "retention": "limits",
  "max_consumers": -1,
  "max_msgs_per_subject": 1,
  "max_msgs": -1,
  "max_bytes": -1,
  "max_age": 0,
  "max_msg_size": -1,
  "storage": "file",
  "discard": "new",
  "num_replicas": 1,
  "duplicate_window": 120000000000,
  "placement": {
    "cluster": ""
  },
  "sources": [
    {
      "name": "KV_A",
      "filter_subject": "$KV.A.>",
      "subject_transform_dest": "$KV.C.>"
    },
    {
      "name": "KV_B",
      "filter_subject": "$KV.B.>",
      "subject_transform_dest": "$KV.C.>"
    }
  ],
  "sealed": false,
  "deny_delete": true,
  "deny_purge": false,
  "allow_rollup_hdrs": true,
  "allow_direct": true,
  "mirror_direct": false
}

(see https://github.com/nats-io/nats.go/pull/1200 for reference)

Clients and Tools

[x] Schemas @jnmoyne
[x] CLI @jnmoyne
[ ] Terraform @ripienaar
[ ] Github Actions @ripienaar
[X] Go @jnmoyne
[X] Java @scottf https://github.com/nats-io/nats.java/pull/978 https://github.com/nats-io/nats.java/pull/982
[X] JavaScript @aricart https://github.com/nats-io/nats.deno/pull/494
[x] .Net @scottf https://github.com/nats-io/nats.net/pull/817 https://github.com/nats-io/nats.net/pull/820
[ ] C @levb
[ ] Python @wallyqs
[ ] Ruby @wallyqs
[ ] Rust @Jarema @caspervonb

Other Tasks

[ ] docs.nats.io updated @bruth
[ ] Update ADR to Implemented
[ ] Update client features spreadsheet

Client authors please update with your progress. If you open issues in your own repositories as a result of this request, please link them to this one by pasting the issue URL in a comment or main issue description.

nats-io / nats-architecture-and-design