search

alibaba / jstorm

Enterprise Stream Process Engine
http://jstorm.io
Apache License 2.0
3.91k stars 1.81k forks source link

KryoTupleDeserializer.deserialize时StackOverflowError #604

Closed sorrytaobao closed 6 years ago

sorrytaobao commented 6 years ago

java.lang.RuntimeException: java.lang.StackOverflowError at backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:103) at backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:65) at backtype.storm.serialization.KryoTupleDeserializer.deserializeBatch(KryoTupleDeserializer.java:116) at com.alibaba.jstorm.task.TaskBatchReceiver$DeserializeBatchRunnable.deserialize(TaskBatchReceiver.java:70) at com.alibaba.jstorm.task.TaskReceiver$DeserializeRunnable.onEvent(TaskReceiver.java:163) at com.alibaba.jstorm.task.TaskBatchReceiver$DeserializeBatchRunnable.onEvent(TaskBatchReceiver.java:46) at backtype.storm.utils.DisruptorQueueImpl.consumeBatchToCursor(DisruptorQueueImpl.java:191) at backtype.storm.utils.DisruptorQueueImpl.consumeBatchWhenAvailable(DisruptorQueueImpl.java:159) at com.alibaba.jstorm.task.TaskReceiver$DeserializeRunnable.run(TaskReceiver.java:187) at com.alibaba.jstorm.task.TaskBatchReceiver$DeserializeBatchRunnable.run(TaskBatchReceiver.java:46) at com.alibaba.jstorm.callback.AsyncLoopRunnable.run(AsyncLoopRunnable.java:95) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.StackOverflowError at java.util.AbstractSet.(AbstractSet.java:63) at java.util.HashSet.(HashSet.java:102) at net.sf.json.AbstractJSON$CycleSet.initialValue(AbstractJSON.java:38) at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:160) at java.lang.ThreadLocal.get(ThreadLocal.java:150) at net.sf.json.AbstractJSON$CycleSet.getSet(AbstractJSON.java:42) at net.sf.json.AbstractJSON.getCycleSet(AbstractJSON.java:284) at net.sf.json.AbstractJSON.addInstance(AbstractJSON.java:63) at net.sf.json.JSONObject._fromBean(JSONObject.java:655) at net.sf.json.JSONObject.fromObject(JSONObject.java:172) at net.sf.json.JSONSerializer.toJSON(JSONSerializer.java:108) at net.sf.json.AbstractJSON._processValue(AbstractJSON.java:238) at net.sf.json.JSONArray._processValue(JSONArray.java:2513) at net.sf.json.JSONArray.processValue(JSONArray.java:2538)

请问这个是什么场景下会用到deserialize，并且什么场景下回调用net.sf.json.JSON相关方法？ 经测试发现某一条数据会出现这种情况，并且100%必现，导致spout一直fail

sorrytaobao commented 6 years ago

经调查发现，这个方法应该是用在消息网络传输上才使用的，如果一个消息需要从集群中的A机器传到B机器，就需要序列化和反序列化，目前就是反序列化的时候出问题了。

sorrytaobao commented 6 years ago

问题自己找到了，用了net.sf.json的转化，序列化的时候没问题，反序列化的时候因为数据里面存在一个null值，导致无限递归，栈溢出了。应该是反序列化和net.sf.json不兼容，又或者是net.sf.json的一个BUG。换成fastjson解决问题。