Closed parbatrajpurohit closed 2 weeks ago
Hi!
Did you try with other examples? Just to discard it is not an issue with the example job
Upon troubleshooting, I discovered why the application keeps running in the Spark Master Web UI, even though it shows as completed on the Spark Connect server.
The issue arises because the Spark Master Web UI treats the Spark Connect server as an application that runs indefinitely until we manually stop or kill the Spark Connect Docker container.
Is there a way to prevent this, so that only the applications I provide from my IDE appear in the Master Web UI and go through the Spark Connect server? This way, there will be a single connection point for the full stack through the Spark Connect server on port 15002 and spark connect server will give task to the Spark-Master.
Hi,
It is not clear to me that this is an issue related with the Bitnami packaging of Spark or with the Spark application itself. Did you try showing this case to the upstream Spark devs?
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.
Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.
Name and Version
bitnami/spark:3.5.1
What architecture are you using?
None
What steps will reproduce the bug?
Here's a Docker Compose setup for a distributed Apache Spark environment using Bitnami's Spark image. It includes:
All services are connected via a custom network and use a shared volume for data.
Once I submit a task to the Spark Connect server on port 15002 from my local machine, the Spark master distributes the workload to the workers. After some time, I can see the output in the PyCharm console. However, the application continues to run on the master, and the workers keep processing it.
To resolve this, I need to manually kill the application. If I try to run a new application after this, I encounter a gRPC error with a status code of 2, indicating an unknown error.
Python Code:
What is the expected behavior?
The expected behavior is for the Spark master to coordinate the distribution of tasks to the Spark workers, and the Spark Connect server to allow for the submission of tasks from your local machine.
However, there are a few issues and points to address to ensure the system functions correctly and prevents the gRPC error with status code 2.
Also once the application is completed in pycharm and it shows the result in the pycharm console it should show the task in list of completed application on the WebUI too
What do you see instead?
Additional information
No response