Open sirajulm opened 1 month ago
Hello @sirajulm, Thanks for finding the time to report the issue! We really appreciate the community's efforts to improve Apache Kyuubi.
This is because Kyuubi does not implement a Spark Standalone ApplicationOperation yet.
For interactive sessions (this is not your case), Kyuubi submits a custom Spark app that has a built-in RPC Server, so that the Kyuubi server can communicate to the Spark driver directly to know the query status.
For batch Spark jobs(no matter Java/Scala jar or PySpark or R), Kyuubi just assembles the spark-submit
command to submit your job to the Resource Manager(Local, Mesos, YARN, K8s, or Standalone), and then uses ApplicationOperation
to query the job status. Currently, it only supports Local, YARN, and K8s.
@pan3793 Thanks for the quick response. I will try with k8s which would match closely to a production setup for me.
Code of Conduct
Search before asking
Describe the bug
I'm trying to submit a pyspark batch job using the REST endpoint. I can see the spark job to be completed, but Kyuubi shows the job is in pending state.
Setup:
Sample pyspark code:
Spark job log
Conclusion From the logs it is clear that the job is being executed as expected. But Kyuubi is not able to find the job status and shows in a forever PENDING state
Affects Version(s)
1.9.1
Kyuubi Server Log Output
Kyuubi Engine Log Output
Kyuubi Server Configurations
Kyuubi Engine Configurations
No response
Additional context
I'm new to Kyuubi and Scala, I would still be happy to support.
Are you willing to submit PR?