apache / rocketmq

Apache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
https://rocketmq.apache.org/
Apache License 2.0
21.19k stars 11.67k forks source link

DEADLINE_EXCEEDED: deadline exceeded after 29.999577300s #6496

Closed charlienss closed 1 year ago

charlienss commented 1 year ago

I used this code:

public class SimpleConsumerExample {
    private static final Logger log = LoggerFactory.getLogger(SimpleConsumerExample.class);

    private SimpleConsumerExample() {
    }

    @SuppressWarnings({"resource", "InfiniteLoopStatement"})
    public static void main(String[] args) throws ClientException {
        final ClientServiceProvider provider = ClientServiceProvider.loadService();

        // Credential provider is optional for client configuration.
        String accessKey = "rocketmq2";
        String secretKey = "12345678";
        SessionCredentialsProvider sessionCredentialsProvider =
            new StaticSessionCredentialsProvider(accessKey, secretKey);

        String endpoints ="10.15.25.4:30402";
        ClientConfiguration clientConfiguration = ClientConfiguration.newBuilder()
                .setRequestTimeout(Duration.ofSeconds(30))
            .setEndpoints(endpoints)
            .setCredentialProvider(sessionCredentialsProvider)
            .build();
        String consumerGroup = "yourConsumerGroup";
        Duration awaitDuration = Duration.ofSeconds(30);
        String tag = "yourMessageTagA";
        String topic = "testTopic";
        FilterExpression filterExpression = new FilterExpression(tag, FilterExpressionType.TAG);
        SimpleConsumer consumer = provider.newSimpleConsumerBuilder()
            .setClientConfiguration(clientConfiguration)
            // Set the consumer group name.
            .setConsumerGroup(consumerGroup)
            // set await duration for long-polling.
            .setAwaitDuration(awaitDuration)
            // Set the subscription for the consumer.
            .setSubscriptionExpressions(Collections.singletonMap(topic, filterExpression))
            .build();
        // Max message num for each long polling.
        int maxMessageNum = 16;
        // Set message invisible duration after it is received.
        Duration invisibleDuration = Duration.ofSeconds(15);
        // Receive message, multi-threading is more recommended.
        do {
            final List<MessageView> messages = consumer.receive(maxMessageNum, invisibleDuration);
            log.info("Received {} message(s)", messages.size());
            for (MessageView message : messages) {
                final MessageId messageId = message.getMessageId();
                try {
                    consumer.ack(message);
                    log.info("Message is acknowledged successfully, messageId={}", messageId);
                } catch (Throwable t) {
                    log.error("Message is failed to be acknowledged, messageId={}", messageId, t);
                }
            }
        } while (true);
        // Close the simple consumer when you don't need it anymore.
        // consumer.close();
    }
}

Then error is: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 29.999577300s. [remote_addr=/10.15.25.4:30402] at io.grpc.Status.asRuntimeException(Status.java:539) at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:544) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563) at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

francisoliverlee commented 1 year ago

这个是5.0客户端问题, 建议issue放:https://github.com/apache/rocketmq-clients/issues

charlienss commented 1 year ago

It's not a 5.0 client issue. This error was returned by the proxy, and it should be a bug in the proxy.

It's not a 5.0 client issue. This error was returned by the proxy, and it should be a bug in the proxy.

xdkxlk commented 1 year ago

You can set useEndpointPortFromRequest to true and try again. See more information in #6268

charlienss commented 1 year ago

I've already set it up,then I can send messages, but I can't consume

I've already set useEndpointPortFromRequest =true,then I can send messages, but I can't consume

francisoliverlee commented 1 year ago

这个错误是GRPC client访问proxy超时了, 我看了下rocketmq client关于deadline的的超时时间设置代码, 应该是deadline= awaitDuration + request timeout, 从你的代码看应该是60s,但是实际报错是30s。

image image

建议检查下

  1. pod网络策略、安全组这种配置
  2. 确认下代码和报错是否匹配,再执行下,现在的报错知道是grpc遇到deadline了,看看能否把rocketmq client相关的堆栈打出来。
  3. 也可以用https://arthas.aliyun.com/doc/trace.html trace下客户端和proxy, 看看请求到proxy没有, 到了的话是真的grpc超时了, 还是长轮训超时了
charlienss commented 1 year ago

这个问题是我用高版本JDK(java11)编译,低版本Jdk(java8)运行造成的,有些方法缺失造成的bug,换成JDK8编译后部署就可以了

miwucc commented 4 days ago

I have the problem too, When i use rocketmq-client-java 5.0.6,5.0.7 同connect rockert proxt 5.3.1, when fetchTopicRoute it happend this problem:

org.apache.rocketmq.shaded.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2.999828700s. [remote_addr=/192.168.110.224:31831]

I am sure the 192.168.110.224:31831 tcp network can connect, beacuse i can telnet it. but the rocketmqclient connect make the grpc exception.

3

the client runtime enriment is jdk17,rocket proxy server is open jdk8

miwucc commented 4 days ago

这个问题是我用高版本JDK(java11)编译,低版本Jdk(java8)运行造成的,有些方法缺失造成的bug,换成JDK8编译后部署就可以了

你用jdk11编译。用jdk8运行,指的是proxy端吗?但是我用的官方5.3.1镜像也报这个错