Closed deshpandeanoop closed 3 years ago
Could you please choose a Spark version greater than 2.4 or 2.4+?
@yanghua : Is there anyway to have this working with spark version 2.3.1
? Currently I'm trying to launch on my local machine for testing purposes. Once I'm done with my development, I will be launching it on our on-prem cluster(shared by multiple teams) which is having spark 2.3.1
installed and we don't have any control to upgrade it.
Hi @vinothchandar Can you chime in to answer this question?
@vinothchandar : A gentle reminder to help us out here :)
@deshpandeanoop noticed that you used a different version "spark-avro" % "2.4.4"
rather than your main spark version 2.3.1
A more important matter: it has been discussed long time back where we only support spark 2.4+
Strongly suggest that you make efforts to upgrade Spark in some way. there are important avro updates in newer version. Spark 2.3.1 uses avro 1.7.7 which can be very problematic https://github.com/apache/spark/blob/30aaa5a3a1076ca52439a905274b1fcf498bc562/pom.xml#L142
Hi Team,
I'm getting
java.lang.NoSuchMethodError
exception when I try launching the Spark application in stand alone mode. Exception Trace:Below is my
build.sbt
file:Spark submit command:
Versions of the softwares installed on my system,
2.11
spark-2.3.1-bin-hadoop2.7
Just to add, I'm trying to read a
csv
extract and create ahudi
table out of it, below is the my sample code that gets executed upon launching my spark application.