search

MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform
MIT License
26 stars 11 forks source link

Livy application in Spark, not always port 4040 #243

Closed ghukill closed 5 years ago

ghukill commented 6 years ago

port 4040 is hardcoded into localsettings.py as the port for the Livy application in Spark, but this is only because counting begins at 4040. If, for example, a pyspark shell is running on cluster, than the Livy app will gobble up the next available port like 4041.

It's unlikely that pyspark will be running often, but worth considering and seeing if the Livy application's port can be sniffed out when instantiated.

ghukill commented 5 years ago

If going to loop through possible incrementing ports, would need to add address @ api_base in SparkAppAPIClient.

ghukill commented 5 years ago

Fixed.

localsettings now contains SPARK_APPLICATION_ROOT_PORT, which sets the base port to attempt to set Spark UI. Increments x100 until finds open port, then sets to LivySession.sparkUiUrl. This is used through GUI to populate links to Spark UI, and for SparkAppAPIClient.