luky777
commented
11 months ago So below I simplified code, just to run hud procedure and this is my results: Error with below code I get is: "It's not a Hudi table". If I change line: spark_df_commits = spark.sql("call show_commits(adpdb.hudi_stage_jira, 5)") to: spark_df_commits = spark.sql("call show_commits('adpdb.hudi_stage_jira', 5)") Then I get different error: ParseException: == SQL == call show_commits('adpdb.hudi_stage_jira', 5) ^^^
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import concat_ws, col, split, size, lit
from pyspark.sql.types import StringType, BooleanType, DateType
import pyspark.sql.functions as F
args = getResolvedOptions(sys.argv, ["JOB_NAME"])
spark = (SparkSession.builder.config('spark.serializer','org.apache.spark.serializer.KryoSerializer') \
.config('spark.sql.hive.convertMetastoreParquet','false') \
.config('spark.sql.catalog.spark_catalog', 'org.apache.spark.sql.hudi.catalog.HoodieCatalog') \
.config('spark.sql.extensions','org.apache.spark.sql.hudi.HoodieSparkSessionExtension') \
.config('spark.sql.legacy.pathOptionBehavior.enabled', 'true').getOrCreate())
sc = spark.sparkContext
glueContext = GlueContext(sc)
job = Job(glueContext)
job.init(args["JOB_NAME"], args)
# Execute the query and show the results
spark_df_commits = spark.sql("call show_commits(adpdb.hudi_stage_jira, 5)")
spark_df_commits.show()
job.commit()
ad1happy2go
Describe the problem you faced Hi, I Can't get Procedures working in Glue version 4. Getting error: py4j.protocol.Py4JJavaError: An error occurred while calling o96.sql. : java.lang.AssertionError: assertion failed: It's not a Hudi table
Here is my Glue script
This is what I have in Error Log:
2023-10-10 14:00:56,574 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(77)): Error from Python:Traceback (most recent call last): File "/tmp/Hudi_CLI_Test.py", line 89, in