moomindani
commented
2 years ago Does the issue happen on Glue's official docker image v3? https://aws.amazon.com/blogs/big-data/develop-and-test-aws-glue-version-3-0-jobs-locally-using-a-docker-container/
Hi! I have installed a local environment on ec2 but every time that I try to run a sql write query it returns a weird java error. It does not happen when I try to perform only selects. I suspect that there are some jar files incompatible between them. Here is the python code:
spark.sql("SELECT * from tmp_tst_roni_athena_save")And here is the error:
Py4JJavaError Traceback (most recent call last) /tmp/ipykernel_4893/213554199.py in
4 AS (SELECT * from tmp_tst_roni_athena_save)
5
----> 6 """)
/opt/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3/python/pyspark/sql/session.py in sql(self, sqlQuery) 721 [Row(f1=1, f2='row1'), Row(f1=2, f2='row2'), Row(f1=3, f2='row3')] 722 """ --> 723 return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped) 724 725 def table(self, tableName):
/opt/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in call(self, *args) 1303 answer = self.gateway_client.send_command(command) 1304 return_value = get_return_value( -> 1305 answer, self.gateway_client, self.target_id, self.name) 1306 1307 for temp_arg in temp_args:
/opt/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3/python/pyspark/sql/utils.py in deco(*a, kw) 109 def deco(*a, *kw): 110 try: --> 111 return f(a, kw) 112 except py4j.protocol.Py4JJavaError as e: 113 converted = convert_exception(e.java_exception)
/opt/spark-3.1.1-amzn-0-bin-3.2.1-amzn-3/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name) 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". --> 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError(
The list of JAR files: