search

kubeflow / spark-operator

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Apache License 2.0
2.79k stars 1.37k forks source link

Expose spark application to the outside world #959

Open tcassaert opened 4 years ago

tcassaert commented 4 years ago

I am currently trying to run our Spark application with a REST API on Kubernetes, using the Spark Operator.

I got the driver and executors up and running, so that's all working. The Spark application is using a Gunicorn, to provide access to it. This gunicorn runs on a random port. This port, however, seems unknown to Kubernetes itself or to the spark operator, or the driver.

Our Spark/Yarn setup makes use of Zookeeper to register the ip and port, but if possible, I'd be nice to have this functionality with Kubernetes itself. Is there any way to make this port known to Kubernetes, so it can be served with a Kubernetes service object?

github-actions[bot] commented 1 week ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.