apache / dubbo

The java implementation of Apache Dubbo. An RPC and microservice framework.
https://dubbo.apache.org/
Apache License 2.0
40.48k stars 26.43k forks source link

Kryo serialization fails(服务既是生产者又是消费者出现问题) #2881

Closed llx206 closed 3 years ago

llx206 commented 5 years ago

https://github.com/apache/incubator-dubbo/issues/2179#issue-347287850

传输的数据较大,想使用kryo序列化,来提高性能。请问,在最新的2.6.5版本下,还是会出现 image

请问大神们,这个问题要如何解决呢,谢谢!!!!

eoc2015 commented 5 years ago

感觉这个可以参考:https://www.2cto.com/kf/201805/743957.html

llx206 commented 5 years ago

@eoc2015 ,是要修改dubbo的代码,才能解决吗?

cvictory commented 5 years ago

I find the how it is happen. I will commit a pr to complete the kryo serialization. But if you want to control the registry class , there will be some constraint to use.

llx206 notifications@github.com 于2018年12月6日周四 下午5:31写道:

@eoc2015 https://github.com/eoc2015 ,是要修改dubbo的代码,才能解决吗?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-dubbo/issues/2881#issuecomment-444807244, or mute the thread https://github.com/notifications/unsubscribe-auth/AD5mbFEyKQDI5IDThVzoS9vXVpbHl20Dks5u2OPkgaJpZM4ZADEY .

--

Best Regard! cvitory

cvictory commented 5 years ago

kryo类型查找会有两种形式,一种是根据类名,一种是根据id和类名的映射。 如果只是根据类名进行映射,不会出现问题,性能稍差;如果想根据id和类名的映射,性能较好,但是id和class的关系,必须保证在server端和client端的顺序一致。 在dubbo中,当你使用optimizer="org.apache.dubbo.benchmark.serialize.SerializationOptimizerImpl" 配置时,就会针对SerializationOptimizerImpl中定义的class开启id和class映射。代码如下: ```

public Collection getSerializableClasses() { return Arrays.asList(User.class, Page.class, UserService.class); }

``` 整个调用出现问题的原理可以看下图。 image

所以解决办法有两种: 1.不要启用这个优化,也就是不要设置optimizer。 2.dubbo代码进行了初步的优化,进行了初步的测试,可以运行。原理如图:

image

原来的接口里有一个全局的映射表,这个逻辑不改变的前提下;针对部分接口维度的优化,可以新建一个id和class列表。这样可以实现功能,但是同时带来了新建和维护这个列表的成本。

victory shenglicao2@gmail.com 于2018年12月12日周三 下午3:04写道:

I find the how it is happen. I will commit a pr to complete the kryo serialization. But if you want to control the registry class , there will be some constraint to use.

llx206 notifications@github.com 于2018年12月6日周四 下午5:31写道:

@eoc2015 https://github.com/eoc2015 ,是要修改dubbo的代码,才能解决吗?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-dubbo/issues/2881#issuecomment-444807244, or mute the thread https://github.com/notifications/unsubscribe-auth/AD5mbFEyKQDI5IDThVzoS9vXVpbHl20Dks5u2OPkgaJpZM4ZADEY .

--

Best Regard! cvitory

--

Best Regard! cvitory

llx206 commented 5 years ago

@cvictory ,非常感谢你细致的回复。还有两个问题再请教下。 1.不要启用这个优化,也就是不要设置optimizer。 问:不启用优化,相比hessian2序列化协议,kryo的性能如何呢? 2.dubbo代码进行了初步的优化,进行了初步的测试,可以运行 问:针对部分接口维度的优化,新建一个ID和class列表,这个会在dubbo的哪个版本加入呢?

期待你的再次回复,谢谢!!

cvictory commented 5 years ago

抱歉,上周休假了,网络实在是惨不忍睹。 问题1,我们没做过测试,部分数据来源于网络:https://x-rip.iteye.com/blog/1555293 。 同时强调一点,如果不用注册方式的话,尽量让报名和类名短,来自于:https://github.com/EsotericSoftware/kryo#optional-registration

问题2,预计要到2.6.6版本中支持。 我还在考虑是否可以考虑使用一种比较快速的方式来解决。 你可以在上面看到我提交的pr。


Sorry, the vacation last week, the network is really terrible. Question 1, we have not tested, some of the data comes from the network: https://x-rip.iteye.com/blog/1555293. At the same time, it is emphasized that if you do not need to register, try to make the registration and class name short, from: https://github.com/EsotericSoftware/kryo#optional-registration.

Question 2 is expected to be supported in version 2.6.6. I am still considering whether I can consider using a faster way to solve it. You can see the pr I submitted above.

llx206 commented 5 years ago

@cvictory 非常感谢你的再次解答。同时,也非常期待你们的新版本能用更快速的方式实现,期待新版本。谢谢!!!