Closed colehudson closed 6 years ago
Just realized, this issue might come into play: https://github.com/WSULib/combine/issues/243.
If another application -- likely another Spark app -- gobbles up 4040
, then Livy will go on to the next port and create the app there, e.g. 4041
.
It's unlikely, but possible. Wonder if we'll want to see if we can pass a preferred port from Livy, e.g. 4066
that would give a comfortable gap between the auto incrementing 4040
, and have reverse proxies point to that.
Nope, not gonna happen. Per https://issues.apache.org/jira/browse/SPARK-20044. Thanks to @ghukill for testing this. Basically, the apps that were going to be proxied through are not designed to accommodate this. Instead, a more sustainable solution is to have these apps firewalled off to the world and access be granted to needed IPs.
Beyond the basic firewall rules one can put in place, we would like to tie all proxied services into a single authentication mechanism. Whether or not that is basic auth through NGINX or something more complicated like authenticating through Django's auth process, we'll see...