Closed ghukill closed 5 years ago
If going to loop through possible incrementing ports, would need to add address @ api_base
in SparkAppAPIClient
.
Fixed.
localsettings
now contains SPARK_APPLICATION_ROOT_PORT
, which sets the base port to attempt to set Spark UI. Increments x100 until finds open port, then sets to LivySession.sparkUiUrl
. This is used through GUI to populate links to Spark UI, and for SparkAppAPIClient
.
port
4040
is hardcoded intolocalsettings.py
as the port for the Livy application in Spark, but this is only because counting begins at4040
. If, for example, a pyspark shell is running on cluster, than the Livy app will gobble up the next available port like4041
.It's unlikely that pyspark will be running often, but worth considering and seeing if the Livy application's port can be sniffed out when instantiated.