[Bug] In high-concurrency scenarios, the connection is interrupted abnormally

AlbinChang commented 1 month ago

Pre-check

[X] I am sure that all the content I provide is in English.

Search before asking

[X] I had searched in the issues and found no similar issues.

Apache Dubbo Component

Java SDK (apache/dubbo)

Dubbo Version

3.2.14

Steps to reproduce this issue

my test code client code

package org.example.client;

import org.apache.dubbo.config.ApplicationConfig;
import org.apache.dubbo.config.ConsumerConfig;
import org.apache.dubbo.config.MethodConfig;
import org.apache.dubbo.config.ReferenceConfig;
import org.apache.dubbo.config.bootstrap.DubboBootstrap;
import org.apache.dubbo.config.bootstrap.builders.ReferenceBuilder;
import org.apache.dubbo.remoting.Constants;
import org.apache.dubbo.rpc.FutureContext;
import org.example.api.GreetingsService;

import java.io.IOException;
import java.util.concurrent.*;

public class Application {

    private static final ExecutorService worker = Executors.newFixedThreadPool(1, (r) ->{
        Thread thread = new Thread(r);
        thread.setName("worker-" + thread.getId());
        return thread;
    } );

    private static final ExecutorService printer = Executors.newFixedThreadPool(1, (r) ->{
        Thread thread = new Thread(r);
        thread.setName("printer-"+thread.getId());
        return thread;
    });

    public static void main(String[] args) throws IOException, InterruptedException {

        ApplicationConfig application = new ApplicationConfig();
        application.setName("client");

        int requests = args.length > 0 ? Integer.parseInt(args[0]) : 100000;
        String ip = args.length > 1 ? args[1] : "localhost";

        MethodConfig methodConfig = new MethodConfig();
        methodConfig.setName("sayHi");
        methodConfig.setAsync(true);
        methodConfig.setTimeout(1000000);

        MethodConfig resetMethodConfig = new MethodConfig();
        resetMethodConfig.setName("reset");
        resetMethodConfig.setAsync(false);

        ConsumerConfig consumerConfig = new ConsumerConfig();
        consumerConfig.setCorethreads(1);
        consumerConfig.setQueues(10000000);
        consumerConfig.setTimeout(1000000);
        consumerConfig.setThreads(1);

        ReferenceConfig<GreetingsService> reference =
                ReferenceBuilder.<GreetingsService>newBuilder()
                .interfaceClass(GreetingsService.class)
                .url("tri://"+ip+":50052")
                        .addMethod( methodConfig )
                        .addMethod( resetMethodConfig )
                        .consumer(consumerConfig)
                        .appendParameter(Constants.CONNECT_TIMEOUT_KEY, "1000000")
                .build();
        DubboBootstrap.getInstance()
                .application(application)
                .reference(reference)
                .start();
        GreetingsService service = reference.get();

        service.reset(); //同步调用，重置服务端的计数器为0

        CountDownLatch latch = new CountDownLatch(requests); // 创建一个计数器，用来记录请求的次数

        int[] count = new int[1];
        long start = System.currentTimeMillis();

        for (int i = 0; i < requests; i++) {

            service.sayHi("dubbo"); //这里的返回值为空，请不要使用
            CompletableFuture<String> future = FutureContext.getContext().getCompletableFuture();
            worker.submit(() -> {
                try {
                    String s = future.get();
                    latch.countDown();

                    printer.submit(() -> {
//                        if( ++count[0] % 1000 == 0 ) {
                            System.out.println("Receive result ======> " + s);
//                        }
                    });
                } catch (InterruptedException e) {
                    throw new RuntimeException(e);
                } catch (ExecutionException e) {
                    throw new RuntimeException(e);
                }
            });
        }

        try {
            latch.await();
        }catch (InterruptedException e)
        {
            e.printStackTrace();
        }

        long end = System.currentTimeMillis();
        long delta = end - start;

        Thread.sleep(1000);

        System.out.println("总计" + requests + " 次");
        System.out.println("耗时" + delta + " 毫秒");
        System.out.println("TPS: " + requests * 1000L /delta + " 次/秒");

        System.exit(0);
    }

}

server code

package org.example.provider;

import org.apache.dubbo.config.ApplicationConfig;
import org.apache.dubbo.config.bootstrap.DubboBootstrap;
import org.apache.dubbo.config.bootstrap.builders.ProtocolBuilder;
import org.apache.dubbo.config.bootstrap.builders.ServiceBuilder;
import org.example.api.GreetingsService;

public class Application {
    public static void main(String[] args) {

        ApplicationConfig application = new ApplicationConfig();
        application.setName("provider");

        DubboBootstrap.getInstance()
                .application(application)
                .protocol(ProtocolBuilder.newBuilder()
                        .dispatcher("all")
                        .accepts(1000000)
                        .queues(1000000)
                        .threadpool("fixed")
                        .threads(1)
                        .name("tri")
                        .port(50052)
                        .build())
                .service(
                        ServiceBuilder.newBuilder()
                                .interfaceClass(GreetingsService.class)
                                .ref(new GreetingsServiceImpl())
                                .build())
                .start()
                .await();
    }
}

api code


package org.example.api;

public interface GreetingsService {
    String sayHi(String name);

    String reset();
}

api impl code


package org.example.provider;

import org.example.api.GreetingsService;

public class GreetingsServiceImpl implements GreetingsService {

    private int count = 0;
//    private AtomicInteger count = new AtomicInteger(0);

    @Override
    public String sayHi(String name) {
        return "hi, " + ++count + "次 " + name;
    }

    @Override
    public String reset() {
        count = 0;
        return "reset success";
    }

}

server 192.168.253.176 startup commad java -cp Dubbo3Test-1.0-SNAPSHOT.jar:./lib/* org.example.provider.Application -Djava.net.preferIPv4Stack=true -Dio.netty.leakDetectionLevel=advanced -Xmx2048m -Xms2048m

client test startup command java -Djava.net.preferIPv4Stack=true -Ddubbo.application.qos-port=33333 -Ddubbo.application.qos-enable=false -Dio.netty.leakDetectionLevel=advanced -Xmx2048m -Xms2048m -cp Dubbo3Test-1.0-SNAPSHOT.jar:lib/* org.example.client.Application 100000 192.168.253.176

The VM specifications for both the client and server are 2C4G

tcp settings

root@ubuntu01:/home/zwb# sysctl -p
net.ipv4.tcp_rmem = 4096 832256  13320192
net.core.rmem_default = 2129920
net.core.rmem_max = 13320192
net.ipv4.tcp_mem = 44379 208128 524288
net.core.wmem_max = 13320192
net.ipv4.tcp_wmem = 4096 832256 13320192

ERROR occurs on the client. Procedure

09:40:35.711 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.711 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.711 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions. 
09:40:35.712 |-ERROR [DubboClientHandler-thread-1] ChainBuilder$CallbackRegistrationInvoker:    -|  [DUBBO] Exception occurred while executing the 0 filter named RpcExceptionFilter., dubbo version: 3.2.14, current host: 192.168.253.210, error code: 2-19. This may be caused by the custom filter is abnormal, go to https://dubbo.apache.org/faq/2/19 to find instructions.

WARN occurs on the server. Procedure

09:40:23.588 |-INFO  [NettyServerWorker-5-1] ing.transport.netty4.NettyChannelHandler:56  -|  [DUBBO] The connection of /192.168.253.201:44932 -> /192.168.253.176:50052 is established., dubbo version: 3.2.14, current host: 192.168.253.193
09:40:32.918 |-INFO  [NettyServerWorker-5-1] ting.transport.netty4.NettyServerHandler:93  -|  [DUBBO] The connection of /192.168.253.201:44932 -> /192.168.253.176:50052 is disconnected., dubbo version: 3.2.14, current host: 192.168.253.193
09:40:32.919 |-WARN  [NettyServerWorker-5-1] .dubbo.remoting.transport.AbstractServer:    -|  [DUBBO] All clients has disconnected from /192.168.253.176:50052. You can graceful shutdown now., dubbo version: 3.2.14, current host: 192.168.253.193, error code: 99-0. This may be caused by unknown error in remoting module, go to https://dubbo.apache.org/faq/99/0 to find instructions. 
09:40:32.925 |-INFO  [NettyServerWorker-5-1] ing.transport.netty4.NettyChannelHandler:72  -|  [DUBBO] The connection of /192.168.253.201:44932 -> /192.168.253.176:50052 is disconnected., dubbo version: 3.2.14, current host: 192.168.253.193
09:40:34.988 |-INFO  [NettyServerWorker-5-2] ing.transport.netty4.NettyChannelHandler:56  -|  [DUBBO] The connection of /192.168.253.201:36978 -> /192.168.253.176:50052 is established., dubbo version: 3.2.14, current host: 192.168.253.193

What you expected to happen

The client program doesn't end as expected and keeps getting stuck

Anything else

No response

Are you willing to submit a pull request to fix on your own?

[X] Yes I am willing to submit a pull request on my own!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

AlbinChang commented 1 month ago

I debug client in idea , this is the exception that throw a Http2Exception (io.netty.handler.codec.http2.Http2Exception: Incomplete header block fragment.

AlbinChang commented 1 month ago

I debug client code in IDEA , go one step further, when state != READ_HEADER_REPRESENTATION. throwing connectionError(COMPRESSION_ERROR, "Incomplete header block fragment.");

wcy666103 commented 1 month ago

Can you submit a pr to help us solve this problem.

Chenjp commented 1 month ago

for v3.2.15-snapshot, unable to reproduce in my local PC with both client and server.

AlbinChang commented 1 month ago

for v3.2.15-snapshot, unable to reproduce in my local PC with both client and server. server is deployed on my Virtual machin， As shown in the following picture Linux version 6.8.0-36-generic (buildd@lcy02-amd64-077) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-23ubuntu4) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #36-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 10 10:49:14 UTC 2024

apache / dubbo