search

apache / dubbo

The java implementation of Apache Dubbo. An RPC and microservice framework.
https://dubbo.apache.org/
Apache License 2.0
40.48k stars 26.43k forks source link

3.2.0版本中序列化问题 #12209

Closed C20171008S closed 1 year ago

C20171008S commented 1 year ago

JDK8 Dubbo:3.2.0 为什么以下两种情况都会解码失败 1.service.exec(new HashMap<String,Object>() {{ put("flag", 1); }}); // 失败

Map<String,Object> params.... params.put("flag",1); service.exec(params); //可以成功 2.service.deleteBy(new Long[]{1L});

C20171008S commented 1 year ago

JDK8 Dubbo:3.2.0 为什么以下两种情况都会解码失败 1.service.exec(new HashMap<String,Object>() {{ put("flag", 1); }}); // 失败

Map<String,Object> params.... params.put("flag",1); service.exec(params); //可以成功 2.service.deleteBy(new Long[]{1L});

补充一下，调整序列化为hessian2是没有问题的

AlbumenJ commented 1 year ago

报错堆栈麻烦贴一下

C20171008S commented 1 year ago

报错堆栈麻烦贴一下


public class Consumer {
private static String zookeeperHost = System.getProperty("zookeeper.address", "127.0.0.1");
public static void main(String[] args) throws InterruptedException {
    ReferenceConfig<Provider.UserService> reference = new ReferenceConfig<>();

    ApplicationConfig application = new ApplicationConfig("dubbo-consumer");
    application.setQosEnable(false);
    reference.setApplication(application);

    reference.setRegistry(new RegistryConfig("zookeeper://" + zookeeperHost + ":2181"));
    reference.setInterface(Provider.UserService.class);
    Provider.UserService service = reference.get();
    CountDownLatch countDownLatch = new CountDownLatch(64);
    long t1 = System.currentTimeMillis();
    for (int i = 0; i < 64; i++) {
        new Thread(() -> {
            for (int i1 = 0; i1 < 10000; i1++) {
                service.exec(new HashMap<String,Object>() {{
                    put("a", 1);
                }});
                service.deleteBy(new Long[]{1L});
            }
            countDownLatch.countDown();

        }).start();
    }
    countDownLatch.await();
    System.err.println("QPS:" + (64 * 10000 / (System.currentTimeMillis() - t1) * 1000));

}

}

org.apache.dubbo.rpc.RpcException: Failed to invoke the method deleteBy in the service com.example.dubbo.Provider$UserService. Tried 3 times of the providers [10.88.107.34:20880] (1/1) from the registry 127.0.0.1:2181 on the consumer 10.88.107.34 using the dubbo version 3.2.0. Last error is: Failed to invoke remote method: deleteBy, provider: dubbo://10.88.107.34:20880/com.example.dubbo.Provider$UserService?anyhost=true&application=dubbo-provider&background=false&category=providers,configurators,routers&check=false&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&interface=com.example.dubbo.Provider$UserService&methods=deleteBy,exec&pid=10516&prefer.serialization=fastjson2,hessian2&qos.enable=false&release=3.2.0&service-name-mapping=true&side=provider&sticky=false&unloadClusterRelated=false, cause: org.apache.dubbo.remoting.RemotingException: Fail to decode request due to: RpcInvocation [methodName=deleteBy, parameterTypes=null] at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:115) at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:341) at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331) at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:101) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331) at org.apache.dubbo.rpc.cluster.filter.support.MetricsClusterFilter.invoke(MetricsClusterFilter.java:51) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331) at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:52) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331) at org.apache.dubbo.rpc.cluster.filter.support.ObservationSenderFilter.invoke(ObservationSenderFilter.java:61) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331) at org.apache.dubbo.spring.security.filter.ContextHolderParametersSelectedTransferFilter.invoke(ContextHolderParametersSelectedTransferFilter.java:41) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331) at org.apache.dubbo.rpc.cluster.filter.support.ConsumerClassLoaderFilter.invoke(ConsumerClassLoaderFilter.java:40) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331) at org.apache.dubbo.rpc.cluster.filter.support.ConsumerContextFilter.invoke(ConsumerContextFilter.java:118) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:194) at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$ClusterFilterInvoker.invoke(AbstractCluster.java:91) at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:103) at org.apache.dubbo.rpc.cluster.support.wrapper.ScopeClusterInvoker.invoke(ScopeClusterInvoker.java:131) at org.apache.dubbo.registry.client.migration.MigrationInvoker.invoke(MigrationInvoker.java:284) at org.apache.dubbo.rpc.proxy.InvocationUtil.invoke(InvocationUtil.java:57) at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:75) at com.example.dubbo.Provider$UserServiceDubboProxy0.deleteBy(Provider$UserServiceDubboProxy0.java) at com.example.dubbo.Consumer.lambda$main$0(Consumer.java:38) at java.base/java.lang.Thread.run(Thread.java:1623) Caused by: java.util.concurrent.ExecutionException: org.apache.dubbo.remoting.RemotingException: Fail to decode request due to: RpcInvocation [methodName=deleteBy, parameterTypes=null] at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096) at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:208) at org.apache.dubbo.rpc.protocol.AbstractInvoker.waitForResultIfSync(AbstractInvoker.java:286) at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:189) at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:71) at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:51) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:331) at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:194) at org.apache.dubbo.rpc.protocol.ReferenceCountInvokerWrapper.invoke(ReferenceCountInvokerWrapper.java:78) at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invokeWithContext(AbstractClusterInvoker.java:380) at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:81) ... 27 more Caused by: org.apache.dubbo.remoting.RemotingException: Fail to decode request due to: RpcInvocation [methodName=deleteBy, parameterTypes=null] at org.apache.dubbo.remoting.exchange.support.DefaultFuture.doReceived(DefaultFuture.java:219) at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:181) at org.apache.dubbo.remoting.exchange.support.DefaultFuture.received(DefaultFuture.java:169) at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleResponse(HeaderExchangeHandler.java:63) at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:192) at org.apache.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:53) at org.apache.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:62) at org.apache.dubbo.common.threadpool.ThreadlessExecutor$RunnableWrapper.run(ThreadlessExecutor.java:141) at org.apache.dubbo.common.threadpool.ThreadlessExecutor.waitAndDrain(ThreadlessExecutor.java:70) at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:202) ... 36 more

AlbumenJ commented 1 year ago

服务端那边的日志有吗

C20171008S commented 1 year ago

服务端那边的日志有吗

public class Provider {
    private static String zookeeperHost = System.getProperty("zookeeper.address", "127.0.0.1");

    public static void main(String[] args) throws Exception {
        ServiceConfig<UserService> service = new ServiceConfig<>();

        ApplicationConfig application = new ApplicationConfig("dubbo-provider");
        application.setQosEnable(false);
        service.setApplication(application);

        service.setRegistry(new RegistryConfig("zookeeper://" + zookeeperHost + ":2181"));
        service.setInterface(UserService.class);
        service.setRef(new UserServiceImpl());
        service.export();

        System.out.println("Dubbo provider started");

        new CountDownLatch(1).await();
    }

    public static class UserServiceImpl implements UserService {

        @Override
        public String exec(Map<String, Object> params) {
            return "success";
        }

        @Override
        public void deleteBy(Long[] ids) {
            System.out.println(JSON.toJSONString(ids));

        }
    }

    interface UserService {
        /**
         * 按照条件执行
         *
         * @param params
         * @return
         */
        String exec(Map<String, Object> params);

        /**
         * 删除
         *
         * @param ids
         */
        void deleteBy(Long[] ids);
    }

}
[traceId: ] 2023-05-04 09:22:31 [NettyServerWorker-3-1] WARN  org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation -  [DUBBO] Decode rpc invocation failed: array not support input TYPED_ANY -110 [1], offset 1/10, dubbo version: 3.2.0, current host: 10.88.107.34, error code: 4-20. This may be caused by , go to https://dubbo.apache.org/faq/4/20 to find instructions. 
com.alibaba.fastjson2.JSONException: array not support input TYPED_ANY -110 [1], offset 1/10
    at com.alibaba.fastjson2.JSONReaderJSONB.startArray(JSONReaderJSONB.java:1265)
    at com.alibaba.fastjson2.reader.ObjectReaderImplInt64Array.readJSONBObject(ObjectReaderImplInt64Array.java:92)
    at com.alibaba.fastjson2.JSONB.parseObject(JSONB.java:535)
    at org.apache.dubbo.common.serialize.fastjson2.FastJson2ObjectInput.readObject(FastJson2ObjectInput.java:118)
    at org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.drawArgs(DecodeableRpcInvocation.java:261)
    at org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:162)
    at org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcInvocation.decode(DecodeableRpcInvocation.java:102)
    at org.apache.dubbo.rpc.protocol.dubbo.DubboCodec.decodeBody(DubboCodec.java:173)
    at org.apache.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:138)
    at org.apache.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:92)
    at org.apache.dubbo.rpc.protocol.dubbo.DubboCountCodec.decode(DubboCountCodec.java:60)
    at org.apache.dubbo.remoting.transport.netty4.NettyCodecAdapter$InternalDecoder.decode(NettyCodecAdapter.java:103)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:1623)
AlbumenJ commented 1 year ago

@wenshao PTAL

C20171008S commented 1 year ago

@wenshao PTAL

我粘贴的用例是基于JDK20测试的，我们的生产项目是用的JDK8，问题是一样的，我这个用例只是复现

C20171008S commented 1 year ago

@AlbumenJ 我试了试fastjson2的jsonb默认序列化对Long数组是支持的 byte[] bytes = JSONB.toBytes(new Long[]{1L}); Long[] longs = JSONB.parseObject(bytes, Long[].class);

我看Dubbo内部序列化和反序列化有一些参数限制，我试了试加上Feature果然就不行了 去除JSONWriter.Feature.WriteClassName是正常的

byte[] bytes = JSONB.toBytes(new Long[]{1L},JSONWriter.Feature.WriteClassName,
            JSONWriter.Feature.FieldBased,
            JSONWriter.Feature.ErrorOnNoneSerializable,
            JSONWriter.Feature.ReferenceDetection,
            JSONWriter.Feature.WriteNulls,
            JSONWriter.Feature.NotWriteDefaultValue,
            JSONWriter.Feature.NotWriteHashMapArrayListClassName,
            JSONWriter.Feature.WriteNameAsSymbol);
    Long[] longs = JSONB.parseObject(bytes, Long[].class, JSONReader.Feature.UseDefaultConstructorAsPossible,
            JSONReader.Feature.ErrorOnNoneSerializable,
            JSONReader.Feature.IgnoreAutoTypeNotMatch,
            JSONReader.Feature.UseNativeObject,
            JSONReader.Feature.FieldBased);

Connected to the target VM, address: '127.0.0.1:61878', transport: 'socket' Exception in thread "main" com.alibaba.fastjson2.JSONException: array not support input TYPED_ANY -110 [1], offset 1/10 at com.alibaba.fastjson2.JSONReaderJSONB.startArray(JSONReaderJSONB.java:1265) at com.alibaba.fastjson2.reader.ObjectReaderImplInt64Array.readJSONBObject(ObjectReaderImplInt64Array.java:92) at com.alibaba.fastjson2.JSONB.parseObject(JSONB.java:681) at com.example.test.TestCase.main(TestCase.java:24)

wenshao commented 1 year ago

https://oss.sonatype.org/content/repositories/snapshots/com/alibaba/fastjson2/fastjson2/2.0.30-SNAPSHOT/ 问题已修复，请帮忙用2.0.30-SNAPSHOT版本验证，2.0.30版本预计在周末5月7日前发布。

C20171008S commented 1 year ago

https://oss.sonatype.org/content/repositories/snapshots/com/alibaba/fastjson2/fastjson2/2.0.30-SNAPSHOT/ 问题已修复，请帮忙用2.0.30-SNAPSHOT版本验证，2.0.30版本预计在周末5月7日前发布。

Long[]已经好了 Map这种还是不行 byte[] bytes = JSONB.toBytes(new HashMap<String,Object>(){{ put("a", "a");}}, JSONWriter.Feature.WriteClassName, JSONWriter.Feature.FieldBased, JSONWriter.Feature.ErrorOnNoneSerializable, JSONWriter.Feature.ReferenceDetection, JSONWriter.Feature.WriteNulls, JSONWriter.Feature.NotWriteDefaultValue, JSONWriter.Feature.NotWriteHashMapArrayListClassName, JSONWriter.Feature.WriteNameAsSymbol); Long[] longs = JSONB.parseObject(bytes, Long[].class, JSONReader.Feature.UseDefaultConstructorAsPossible, JSONReader.Feature.ErrorOnNoneSerializable, JSONReader.Feature.IgnoreAutoTypeNotMatch, JSONReader.Feature.UseNativeObject, JSONReader.Feature.FieldBased);

wenshao commented 1 year ago

抱歉，没看到回复。

https://oss.sonatype.org/content/repositories/snapshots/com/alibaba/fastjson2/fastjson2/2.0.32-SNAPSHOT/ 请帮忙用2.0.32-SNAPSHOT版本验证，2.0.32版本预计在5月28日发布。

C20171008S commented 1 year ago

抱歉，没看到回复。

https://oss.sonatype.org/content/repositories/snapshots/com/alibaba/fastjson2/fastjson2/2.0.32-SNAPSHOT/ 请帮忙用2.0.32-SNAPSHOT版本验证，2.0.32版本预计在5月28日发布。

这次都好了，后期发现问题我再上fastjson2的仓库里边提缺陷。多谢。