search

astronomy-commons / axs

Astronomy eXtensions for Spark: Fast, Scalable, Analytics of Billion+ row catalogs
https://axs.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
23 stars 12 forks source link

Exception when running crossmatch with Spark 3.0.0 version #27

Closed stargaser closed 3 years ago

stargaser commented 3 years ago

I've installed at IPAC the automated build version that @stevenstetzler mentioned in #26:

Yeah, the automated builds seem to work still. I recently used it to make a new Spark 3 release with an updated version of Scala (I was running into similar issues as @stargaser using Spark 3 with AXS until I updated Scala).

When I run crossmatch, an exception is raised that I think is the same as the one I encountered when I built axs and axs-spark myself from the instructions in #20. Does this mean I need to upgrade Scala, and if so, what version should I have?

The informational part of the exception:

Py4JJavaError: An error occurred while calling o234.crossmatch.
: java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
    at org.dirac.axs.FrameFunctions$.crossmatch(FrameFunctions.scala:42)
    at org.dirac.axs.FrameFunctions.crossmatch(FrameFunctions.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:745)
stargaser commented 3 years ago

I think I found my problem. I had spark.jars in the $SPARK_HOME/conf/spark-defaults.conf file pointing to the wrong jar for AxsUtilities. I can run a crossmatch with a local cluster now.

stargaser commented 3 years ago

Now that I have fixed spark.jars, I can crossmatch. I see that the upgraded Scala is in one of the jar files. I am closing this issue.