push consumer deliver_subject does not match wildcarded subscription

[RoadRunnr]

Observed behavior

A user of a (push) consumer with deliver_subject must first create a subscription for the deliver_subject before creating the consumer to receive message for that consumer. When that subscription uses a wildcard, then messages from the consumer are not delivered.

In the sample trace the client subscribes to _MY_KV_SUB.25rsjW3I9BlkBDBvtcdMhW.* and creates the consumer with "deliver_subject": "_MY_KV_SUB.25rsjW3I9BlkBDBvtcdMhW.1Hz9dd0khEFwXrqEJ8XjEK". The subscription clearly covers the deliver_subject. However, there is nothing delivered to that subject.

Sample trace:

[1] 2024/10/18 09:24:45.381642 [TRC] 127.0.0.1:50124 - cid:6 - <<- [SUB _MY_KV_SUB.25rsjW3I9BlkBDBvtcdMhW.* 2]
[1] 2024/10/18 09:24:45.422179 [TRC] 127.0.0.1:50124 - cid:6 - <<- [PUB $JS.API.CONSUMER.CREATE.KV_TEST_KV _INBOX.2xOnO4Sb5zhWymzfrDAyy5.IvCQT5enGy8 377]
[1] 2024/10/18 09:24:45.422240 [TRC] 127.0.0.1:50124 - cid:6 - <<- MSG_PAYLOAD: ["{\"config\":{\"num_replicas\":1,\"deliver_policy\":\"last_per_subject\",\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"filter_subject\":\"$KV.TEST_KV.>\",\"replay_policy\":\"instant\",\"flow_control\":true,\"idle_heartbeat\":5000000000,\"headers_only\":true,\"deliver_subject\":\"_MY_KV_SUB.25rsjW3I9BlkBDBvtcdMhW.1Hz9dd0khEFwXrqEJ8XjEK\",\"mem_storage\":true},\"stream_name\":\"KV_TEST_KV\"}"]
[1] 2024/10/18 09:24:45.422928 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.consumer_create_response\",\"stream_name\":\"KV_TEST_KV\",\"name\":\"NhGtJcrk\",\"created\":\"2024-10-18T09:24:45.42255032Z\",\"config\":{\"deliver_policy\":\"last_per_subject\",\"ack_policy\":\"none\",\"ack_wait\":79200000000000,\"max_deliver\":1,\"filter_subject\":\"$KV.TEST_KV.\\u003e\",\"replay_policy\":\"instant\",\"idle_heartbeat\":5000000000,\"flow_control\":true,\"headers_only\":true,\"deliver_subject\":\"_MY_KV_SUB.25rsjW3I9BlkBDBvtcdMhW.1Hz9dd0khEFwXrqEJ8XjEK\",\"inactive_threshold\":5000000000,\"num_replicas\":1,\"mem_storage\":true},\"delivered\":{\"consumer_seq\":0,\"stream_seq\":0},\"ack_floor\":{\"consumer_seq\":0,\"stream_seq\":0},\"num_ack_pending\":0,\"num_redelivered\":0,\"num_waiting\":0,\"num_pending\":2,\"ts\":\"2024-10-18T09:24:45.422697365Z\"}"]
[1] 2024/10/18 09:24:45.422989 [TRC] 127.0.0.1:50124 - cid:6 - ->> [MSG _INBOX.2xOnO4Sb5zhWymzfrDAyy5.IvCQT5enGy8 1 735]

There is nothing in the wire protocol or consumer documentation that suggest that wildcarded subscription can not be used on the deliver_subject

Expected behavior

consumer message are delivered to deliver_subject when a wild card subscription for that subject exists.

Server and client version

• nats-server: v2.10.21
• client: raw wire protocol

Host environment

Linux, server in docker container from hub.docker.com, started with sudo podman run --network=host nats -js -DVV

Steps to reproduce

I'm sure the effect can be recreated with plain PUB/SUB messages in any client.

However, the problem was discovered with a new client for Erlang. The reproducer therefore needs Erlang/OTP 27.1 and rebar3.

If you already have a working Erlang/OTP 27.1 and rebar3 installation, run a local nats server, checkout and run the test:

git clone -b broken-wildcard-sub-for-consumer https://github.com/travelping/enats.git
cd enats
rebar3 ct --suite=test/nats_jetstream_SUITE --case=kv

The test run stall/block due to the missing messages from the consumer.

If you need a working Erlang/OTP 27.1 and rebar3 installation, you can use the erlang:27 docker container or install it on a recent Linux manually with something like this:

mkdir ~/bin
export PATH=$PATH:~/bin
curl -o ~/bin/kerl https://raw.githubusercontent.com/kerl/kerl/master/kerl
chmod a+x ~/bin/kerl
kerl update releases
kerl build-install 27.1.2 otp-27 ~/bin/otp-27
. ~/bin/otp-27/activate
curl -o ~/bin/rebar3 https://github.com/erlang/rebar3/releases/download/3.24.0/rebar3
chmod a+x ~/bin/rebar3

[neilalexander]

Have reproduced in a server unit test.

Appears to be because (*Sublist).registerNotification() does equal matching of the subjects for interest updates rather than using Match(), so for a split second at creation, the consumer has interest on the wildcard subscriber, but then the sublist interest notification comes along and resets it.

Trying to figure out if it's that way for a specific reason.

[neilalexander]

I think this behaviour is intentional and dates back to nats-io/nats-server#1705.

In short, if you had wildcard matching of subscribers on the DeliverSubject, then a single client that subscribes to > for example would:

1. Register interest on every push consumer, even those that don't have genuine interest from genuine clients, and cause them to start delivering, which could be a problem if some of them are AckNone or limit redeliveries;
2. Elicit additional new heartbeat notifications from every push consumer, generating potentially a ton of traffic.