Watch KV pair timeout

[GmaD-X]

Version

4.1.1

Context

My application watches a couple of KV pairs in Consul. While booting the application, I can see the below error. I have run it before, and never saw this error. Now its not reading the changes I make to the KV pair.

2022-04-21 13:21:42,057 ERROR [vert.x-eventloop-thread-0] [?:?] - io.vertx.core.http.impl.NoStackTraceTimeoutException: The timeout period of 10000ms has been exceeded while executing GET /v1/kv/XYX%2Fservice%2Flogger%2Fkey?wait=10m&index=123 for server 127.0.0.1:8500 - [] - [] -

My consul client options are as below:

public ConsulClientOptions getConsulClientOptions(ConsulClientConfig consulClientConfig) {
    return new ConsulClientOptions()
            .setHost("localhost")
            .setPort(8500)
            .setSsl(false)
            .setTrustAll(true)
            .setVerifyHost(false)
            .setAclToken("token")
            .setKeepAlive(false)
            .setTimeout(10000);
}

/**
 * To watch key and notify changes
 */
private void startWatchOnKey() {
   Watch<KeyValue> watchLeader = Watch.key(getKey(), this.vertx, getConsulClientOptions());
    watchLeader.setHandler(response -> {
        if (response.succeeded()) {
             // do something
        }
    });
    watchLeader.start();
}

Using a dockerized consul

consul:1.9

Do you have a reproducer?

NO

Steps to reproduce

As said above

Extra

Open JDK 11

I get that this is a blocking query where it is blocked for 10minutes (as per log ) and the timeout I have set in my consul options is of 10000ms (10 seconds).

I am wondering why it is happening all of a sudden? Is it an underlying issue? My KV changes are not being read due to the timeout. How can I fix it without increasing the overall timeout?

[GmaD-X]

Hello, can I get an update on this please

[ruslansennov]

Hi @GmaD-X The Watch uses blocking queries to the consul client with a wait parameter of 10 min (see here). This is expected behavior.

[GmaD-X]

Hi @ruslansennov I understand it is a blocking query and it waits for 10 min but, I was not getting any such errors when I had a lower timeout. I had not seen this error in my application for at least 6 months, since I started using the vertx library.
I started getting this error abruptly, so wondering what could've triggered such a behavior. And it is not consistent as well. At times when I set a lower timeout from consul options, I don't see the watch KV timeout. At times I do. Majority of the times, I am seeing the timeout error lately.

Please note I didn't change the version of vertx during this period as well.

[ruslansennov]

Unfortunately I'm out of ideas :(

[pqab]

Hi @ruslansennov

From the consul blocking queries documentation https://developer.hashicorp.com/consul/api-docs/features/blocking

In addition to index, endpoints that support blocking will also honor a wait parameter specifying a maximum duration for the blocking request. This is limited to 10 minutes. If not set, the wait time defaults to 5 minutes. This value can be specified in the form of "10s" or "5m" (i.e., 10 seconds or 5 minutes, respectively). A small random amount of additional wait time is added to the supplied maximum wait time to spread out the wake up time of any concurrent requests. This adds up to wait / 16 additional time to the maximum duration.

Isn't the wait parameter should be configurable from the client side? Could you share more details why it has to be hard code with 10m?

[ruslansennov]

@ViktoriiaLebedeva take a look please

[ViktoriiaLebedeva]

Ok

[ViktoriiaLebedeva]

The waiting value will be taken from the timeout field of the Consul client options