search

dccspeed / fractal

Apache License 2.0
28 stars 8 forks source link

Problem running examples on Spark mode #23

Closed kygx-legend closed 2 years ago

kygx-legend commented 3 years ago

Hello, we are very interested in "Fractal" and trying to run the examples in the distributed setting. But we encountered a problem when running the example and it is hanging there:

            java.lang.UnsupportedOperationException: Accumulator must be registered before send to executor

Would you mind helping figure out the reason? Thanks a lot! The whole log is here test.log .

maniaabdi commented 3 years ago

I have the same issue when I running fractal on spark distributed mode.

viniciusvdias commented 3 years ago

Hi, my guess is Spark versioning issue. Good catch. Could you try using Spark v2.2.0 (https://archive.apache.org/dist/spark/spark-2.2.0/spark-2.2.0-bin-hadoop2.7.tgz) to make sure?

maniaabdi commented 3 years ago

@viniciusvdias I used the same version of Spark that is in the ReadMe which I guess is the same as Spark2-2 and Hadoop 2.7.

maniaabdi commented 3 years ago

I was able to resolve the issue by disabling the Kryo serializer and use Spark default serializer.