`KeyValue.Watch()` blocks forever for certain values of the `keys` parameter

[mihaitodor]

Observed behavior

The kv.Watch("\"001.>\"") call blocks forever if there's a key starting with 001. in Nats KV.

Additionally, if you also try kv.Watch(""), you get a rather cryptic error: nats: jetstream not enabled.

Expected behavior

kv.Watch() should reject invalid keys or, at the very least, it shouldn't block forever.

Also, kv.Watch("") should return a more informative error message, maybe saying that the keys parameter cannot be set to an empty string.

Server and client version

• nats:2.10.12 Docker image
• github.com/nats-io/nats.go v1.34.1 client libraries
• nats CLI v0.1.4
• Go v1.22.1 (darwin/arm64)

Host environment

OSX Sonoma 14.4.1 on Macbook Pro M3 Max

Steps to reproduce

Run the following shell commands:

> docker run --rm -it -p4222:4222 nats:2.10.12 --js
> nats kv add foobar
> nats kv put foobar 001.123 abc

Create a go.mod file with the following contents:

module test

go 1.22.1

require github.com/nats-io/nats.go v1.34.1

require (
    github.com/klauspost/compress v1.17.2 // indirect
    github.com/nats-io/nkeys v0.4.7 // indirect
    github.com/nats-io/nuid v1.0.1 // indirect
    golang.org/x/crypto v0.18.0 // indirect
    golang.org/x/sys v0.16.0 // indirect
)

And then write the following code in a file called main.go:

package main

import (
    "context"
    "log"
    "time"

    "github.com/nats-io/nats.go"
)

func main() {
    conn, err := nats.Connect("localhost:4222")
    if err != nil {
        log.Fatal(err)
    }

    js, err := conn.JetStream()
    if err != nil {
        log.Fatal(err)
    }

    kv, err := js.KeyValue("foobar")
    if err != nil {
        log.Fatal(err)
    }

    // watcher, err := kv.Watch("001.>")
    watcher, err := kv.Watch("\"001.>\"")
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        if err := watcher.Stop(); err != nil {
            log.Fatal(err)
        }
    }()

    var keys []any
    ctx, done := context.WithTimeout(context.Background(), 3*time.Second)
    defer done()
loop:
    for {
        select {
        case entry := <-watcher.Updates():
            if entry == nil {
                break loop
            }
            keys = append(keys, entry.Key())
        case <-ctx.Done():
            log.Fatal("timeout")
        }
    }
    log.Printf("%v", keys)
}

If you execute go mod tidy && go run main.go, you'll see a message containing timeout. If you uncomment line 27 and comment out line 28, you should see [001.123] instead.

Additionally, if you try kv.Watch(""), you'll get a rather cryptic error: nats: jetstream not enabled.

(cc @codegangsta)

[mihaitodor]

Update: I also investigated switching to the new github.com/nats-io/nats.go/jetstream API and I can still reproduce the issue with the following code:

package main

import (
    "context"
    "log"
    "time"

    "github.com/nats-io/nats.go"
    "github.com/nats-io/nats.go/jetstream"
)

func main() {
    conn, err := nats.Connect("localhost:4222")
    if err != nil {
        log.Fatal(err)
    }

    js, err := jetstream.New(conn)
    if err != nil {
        log.Fatal(err)
    }

    kv, err := js.KeyValue(context.Background(), "foobar")
    if err != nil {
        log.Fatal(err)
    }

    // watcher, err := kv.Watch(ctx, "001.>")
    watcher, err := kv.Watch(context.Background(), "\"001.>\"")
    if err != nil {
        log.Fatal(err)
    }
    defer func() {
        if err := watcher.Stop(); err != nil {
            log.Fatal(err)
        }
    }()

    var keys []any
    ctx, done := context.WithTimeout(context.Background(), 3*time.Second)
    defer done()
loop:
    for {
        select {
        case entry := <-watcher.Updates():
            if entry == nil {
                break loop
            }
            keys = append(keys, entry.Key())
        case <-ctx.Done():
            log.Fatal("timeout")
        }
    }
    log.Printf("%v", keys)
}

I can, however, now pass a context to kv.Watch() which makes it easier to abort after a while, but I still think the kv.Watch() call should reject invalid keys.

Also, the nats: jetstream not enabled error is still returned for kv.Watch(context.Background(), "").

[Jarema]

Thanks for the detailed report!

We will take a look soon.

[piotrpio]

@mihaitodor there were 2 issues:

• The Watch("\"001.>\"") was a server issue, fixed here: https://github.com/nats-io/nats-server/pull/5318
• Watch("") returned wrong error due to lacking client-side validation: #1613

Both will now return an error since both are essentially invalid arguments: nats: invalid key: keys cannot be empty and must be a valid NATS subject.

[mihaitodor]

That's awesome, thank you very much for the quick fixes!

[piotrpio]

Both PRs are merged and will be included in the next releases of nats-server and nats.go.