Creating a stream with an all wildcard subject with 3 or 4 fields via rust client causes unexpected behaviour

[sloveridge]

Observed behavior

Creating a stream via the rust client with a subject of "*.*.*.*" or "*.*.*" causes unexpected behaviour when listing streams via the cli. In addition, it will cause subsequent KV store creation to fail. Streams created with these subjects though the cli do not exhibit this behaviour.

eg: after creating a stream "logs" with subject "*.*.*.*" via the rust client.

nats stream ls

nats: error: could not list streams: server response is not a valid "io.nats.jetstream.api.v1.stream_list_response" message: /type: value must be "io.nats.jetstream.api.v1.stream_list_response"
missing properties: 'error'
/streams: expected array, but got null

The stream itself has been created and shows correctly using info

nats stream info logs

Information for Stream logs created 2024-02-19 21:58:01

              Subjects: *.*.*.*
              Replicas: 1
               Storage: File

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

State:

              Messages: 3
                 Bytes: 3.1 KiB
        First Sequence: 1 @ 2024-02-19 21:58:01 UTC
         Last Sequence: 3 @ 2024-02-19 21:58:06 UTC
      Active Consumers: 0
    Number of Subjects: 2

It looks like the stream is listening to Internal JS messages

nats stream view logs

[1] Subject: $JS.EVENT.ADVISORY.API Received: 2024-02-19T21:58:01+11:00

{"type":"io.nats.jetstream.advisory.v1.api_audit","id":"u65MrVboJq8cVj2CymH3QO","timestamp":"2024-02-19T10:58:01.322457455Z","server":"NA2WJO64RVBIQ4X6ALJBAR5XFN3TCCUCXP2OT43W3DQAJXGO337TEMWX","client":{"start":"2024-02-19T10:58:01.320175297Z","host":"172.17.0.1","id":5,"acc":"$G","lang":"rust","ver":"0.33.0","rtt":464729,"server":"NA2WJO64RVBIQ4X6ALJBAR5XFN3TCCUCXP2OT43W3DQAJXGO337TEMWX","kind":"Client","client_type":"nats"},"subject":"$JS.API.STREAM.CREATE.logs","request":"{\"name\":\"logs\",\"max_bytes\":0,\"max_msgs\":0,\"max_msgs_per_subject\":0,\"discard\":\"old\",\"subjects\":[\"*.*.*.*\"],\"retention\":\"limits\",\"max_consumers\":0,\"max_age\":0,\"storage\":\"file\",\"num_replicas\":0,\"consumer_limits\":null}","response":"{\"type\":\"io.nats.jetstream.api.v1.stream_create_response\",\"config\":{\"name\":\"logs\",\"subjects\":[\"*.*.*.*\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msgs_per_subject\":-1,\"max_msg_size\":-1,\"discard\":\"old\",\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"compression\":\"none\",\"allow_direct\":false,\"mirror_direct\":false,\"sealed\":false,\"deny_delete\":false,\"deny_purge\":false,\"allow_rollup_hdrs\":false,\"consumer_limits\":{}},\"created\":\"2024-02-19T10:58:01.321920901Z\",\"state\":{\"messages\":0,\"bytes\":0,\"first_seq\":0,\"first_ts\":\"0001-01-01T00:00:00Z\",\"last_seq\":0,\"last_ts\":\"0001-01-01T00:00:00Z\",\"consumer_count\":0},\"ts\":\"2024-02-19T10:58:01.322421965Z\",\"did_create\":true}"}

[2] Subject: $JS.API.STREAM.LIST Received: 2024-02-19T21:58:06+11:00

{"offset":0}

...

Attempting to create a KV afterwards via either the rust client or CLI will fail

nats kv add test

nats: error: nats: subjects overlap with an existing stream

Expected behavior

Rust client should act similar to the CLI when creating a stream with a subject of "*.*.*.*"

Server and client version

nats-server: v2.10.11 async-nats: v0.3.3 nats-cli: v0.1.3

Host environment

Server started via docker run -d --rm --name nats-test -p 4222:4222 nats:2.10.11 -js

Steps to reproduce

Stream created with the following will cause the issue:

use async_nats::jetstream::stream;

#[tokio::main]
async fn main() {
    let client = async_nats::connect("localhost:4222").await.unwrap();
    let js = async_nats::jetstream::new(client);
    js.get_or_create_stream(stream::Config {
        name: "logs".to_owned(),
        subjects: vec!["*.*.*.*".to_string()],
        ..Default::default()
    })
    .await
    .unwrap();
}

Only using 2 * will not trigger the behaviour.

use async_nats::jetstream::stream;

#[tokio::main]
async fn main() {
    let client = async_nats::connect("localhost:4222").await.unwrap();
    let js = async_nats::jetstream::new(client);
    js.get_or_create_stream(stream::Config {
        name: "logs".to_owned(),
        subjects: vec!["*.*".to_string()],
        ..Default::default()
    })
    .await
    .unwrap();
}

[Jarema]

Hey!

Thanks for creating the issue.

Your suspicion is correct. Creating such a stream almost breaks the JetStream. This is a know footgun that is hard to protect against it, as users might have a different set of tokens in JS API prefix with imports and exports.

You can actually create such a stream with CLI too. Just delete all the streams you currently have.

[sloveridge]

The approach I am taking now to guard against this is to always add a fixed prefix to the subjects, either the stream name or an abbreviation of it.

e.g: "logs.*.*.*.*"

I assume this is sufficient to prevent this behaviour? Main downside here is wasted storage space and ensuring the prefix is added in the application code.

Definitely a footgun. The first error I had was KV creation not working. Took a while to trace it back to this.

[Jarema]

There are only two ways you can get into trouble like this:

• you create a stream that intercepts $JS.API calls
• you create a stream that intercepts subject at which $JS.API is imported from another account

So yes, this should be sufficient to prevent such behavior.

Sorry that you hit that footgun. We're thinking how to improve that, but that is not obvious with given flexibility of setting up the JS prefixes.

[Jarema]

I'm closing the issue. Feel free to reopen if you want to discuss it furhter. We are open to ideas how to improve that, however that is not Rust client specific thing.