nats-io / natscli

The NATS Command Line Interface
Apache License 2.0
489 stars 96 forks source link

Not recognizing difference in stream configuration for sources #1010

Closed anthonyjacques20 closed 7 months ago

anthonyjacques20 commented 7 months ago

Observed behavior

When changing the sources section of the json configuration of a stream and running nats stream edit {stream_name} --config {json_file}, it returns No difference in configuration

Expected behavior

When changing the sources section of the json configuration, it should recognize the change in configuration and apply it

Server and client version

Server: 2.10.11 CLI: 0.1.3

Host environment

Mac

Steps to reproduce

  1. Start jetstream server
  2. Create regular stream
    
    nats stream add my_stream
    ? Subjects my_stream.>
    ? Storage file
    ? Replication 1
    ? Retention Policy Limits
    ? Discard Policy Old
    ? Stream Messages Limit -1
    ? Per Subject Messages Limit -1
    ? Total Stream Size -1
    ? Message TTL -1
    ? Max Message Size -1
    ? Duplicate tracking time window 2m0s
    ? Allow message Roll-ups No
    ? Allow message deletion Yes
    ? Allow purging subjects or the entire stream Yes
    Stream my_stream was created

Information for Stream my_stream created 2024-03-08 11:28:07

          Subjects: my_stream.>
          Replicas: 1
           Storage: File

Options:

         Retention: Limits
   Acknowledgments: true
    Discard Policy: Old
  Duplicate Window: 2m0s
        Direct Get: true
 Allows Msg Delete: true
      Allows Purge: true
    Allows Rollups: false

Limits:

  Maximum Messages: unlimited

Maximum Per Subject: unlimited Maximum Bytes: unlimited Maximum Age: unlimited Maximum Message Size: unlimited Maximum Consumers: unlimited

State:

          Messages: 0
             Bytes: 0 B
    First Sequence: 0
     Last Sequence: 0
  Active Consumers: 0
3. Create sourced stream

nats stream add sourced_stream --source="my_stream" ? Storage file ? Replication 1 ? Retention Policy Limits ? Discard Policy Old ? Stream Messages Limit -1 ? Total Stream Size -1 ? Message TTL -1 ? Max Message Size -1 ? Duplicate tracking time window 2m0s ? Allow message Roll-ups No ? Allow message deletion Yes ? Allow purging subjects or the entire stream Yes ? Adjust source "my_stream" start No ? Adjust source "my_stream" filter and transform Yes ? Filter source by subject (hit enter to finish) my_stream.hello.> ? Subject transform destination ? Filter source by subject (hit enter to finish) ? Import "my_stream" from a different JetStream domain No ? Import "my_stream" from a different account No Stream sourced_stream was created

Information for Stream sourced_stream created 2024-03-08 11:29:52

                  Replicas: 1
                   Storage: File

Options:

                 Retention: Limits
           Acknowledgments: true
            Discard Policy: Old
          Duplicate Window: 2m0s
                Direct Get: true
         Allows Msg Delete: true
              Allows Purge: true
            Allows Rollups: false

Limits:

          Maximum Messages: unlimited
       Maximum Per Subject: unlimited
             Maximum Bytes: unlimited
               Maximum Age: unlimited
      Maximum Message Size: unlimited
         Maximum Consumers: unlimited

Replication:

                   Sources: my_stream

Source Information:

               Stream Name: my_stream

Subject Filter and Transform: my_stream.hello.> untransformed Lag: 0 Last Seen: never

State:

                  Messages: 0
                     Bytes: 0 B
            First Sequence: 0
             Last Sequence: 0
          Active Consumers: 0
4. Get json config of sourced stream with `nats stream info sourced_stream -j`

{ "config": { "name": "sourced_stream", "retention": "limits", "max_consumers": -1, "max_msgs_per_subject": -1, "max_msgs": -1, "max_bytes": -1, "max_age": 0, "max_msg_size": -1, "storage": "file", "discard": "old", "num_replicas": 1, "duplicate_window": 120000000000, "sources": [ { "name": "my_stream", "subject_transforms": [ { "src": "my_stream.hello.\u003e", "dest": "" } ] } ], "sealed": false, "deny_delete": false, "deny_purge": false, "allow_rollup_hdrs": false, "allow_direct": true, "mirror_direct": false, "consumer_limits": {} }, "created": "2024-03-08T19:29:52.008395Z", "state": { "messages": 0, "bytes": 0, "first_seq": 0, "first_ts": "0001-01-01T00:00:00Z", "last_seq": 0, "last_ts": "0001-01-01T00:00:00Z", "consumer_count": 0 }, "cluster": { "leader": "NDHWEXHYZNUKV4TLJ2USHZ23NXVF6ORKES7WYF3UJYWA3GT2L6DSTSIN" }, "sources": [ { "name": "my_stream", "lag": 0, "active": 437615333, "subject_transforms": [ { "src": "my_stream.hello.\u003e", "dest": "" } ] } ], "ts": "2024-03-08T19:30:30.506977Z" }

5. Modify `sources` section by adding another subject transformation

{ "config": { "name": "sourced_stream", "retention": "limits", "max_consumers": -1, "max_msgs_per_subject": -1, "max_msgs": -1, "max_bytes": -1, "max_age": 0, "max_msg_size": -1, "storage": "file", "discard": "old", "num_replicas": 1, "duplicate_window": 120000000000, "sources": [ { "name": "my_stream", "subject_transforms": [ { "src": "my_stream.hello.\u003e", "dest": "" } ] } ], "sealed": false, "deny_delete": false, "deny_purge": false, "allow_rollup_hdrs": false, "allow_direct": true, "mirror_direct": false, "consumer_limits": {} }, "created": "2024-03-08T19:29:52.008395Z", "state": { "messages": 0, "bytes": 0, "first_seq": 0, "first_ts": "0001-01-01T00:00:00Z", "last_seq": 0, "last_ts": "0001-01-01T00:00:00Z", "consumer_count": 0 }, "cluster": { "leader": "NDHWEXHYZNUKV4TLJ2USHZ23NXVF6ORKES7WYF3UJYWA3GT2L6DSTSIN" }, "sources": [ { "name": "my_stream", "lag": 0, "active": 437615333, "subject_transforms": [ { "src": "my_stream.hello.\u003e", "dest": "" }, { "src": "my_stream.hello_world.\u003e", "dest": "" } ] } ], "ts": "2024-03-08T19:30:30.506977Z" }

6. Try to edit the stream with the new config

nats stream edit sourced_stream --config sourced_stream_config.json No difference in configuration

anthonyjacques20 commented 7 months ago

Closing this as I wasn't modifying the source in the correct place. I should've been modifying it in the config key like this. Sharing in case this helps someone else in the future.

{
    "config": {
      "name": "sourced_stream",
      "retention": "limits",
      "max_consumers": -1,
      "max_msgs_per_subject": -1,
      "max_msgs": -1,
      "max_bytes": -1,
      "max_age": 0,
      "max_msg_size": -1,
      "storage": "file",
      "discard": "old",
      "num_replicas": 1,
      "duplicate_window": 120000000000,
      "sources": [
        {
          "name": "my_stream",
          "subject_transforms": [
            {
              "src": "my_stream.hello.\u003e",
              "dest": ""
            },
            {
              "src": "my_stream.hola.\u003e",
              "dest": ""
            }
          ]
        }
      ],
      "sealed": false,
      "deny_delete": false,
      "deny_purge": false,
      "allow_rollup_hdrs": false,
      "allow_direct": true,
      "mirror_direct": false,
      "consumer_limits": {}
    },
    "created": "2024-03-08T19:29:52.008395Z",
    "state": {
      "messages": 0,
      "bytes": 0,
      "first_seq": 0,
      "first_ts": "0001-01-01T00:00:00Z",
      "last_seq": 0,
      "last_ts": "0001-01-01T00:00:00Z",
      "consumer_count": 0
    },
    "cluster": {
      "leader": "NDHWEXHYZNUKV4TLJ2USHZ23NXVF6ORKES7WYF3UJYWA3GT2L6DSTSIN"
    },
    "sources": [
      {
        "name": "my_stream",
        "lag": 0,
        "active": 437615333,
        "subject_transforms": [
          {
            "src": "my_stream.hello.\u003e",
            "dest": ""
          }
        ]
      }
    ],
    "ts": "2024-03-08T19:30:30.506977Z"
  }