search

gettyimages / docker-spark

Docker build for Apache Spark
MIT License
679 stars 369 forks source link

removed deprecated 'spark.executor.port' from configuration #43

Closed serge-m closed 6 years ago

serge-m commented 6 years ago

Trigger

Existing config generates a warning 

The configuration key 'spark.executor.port' has been deprecated as of Spark 2.0.0 and may be removed in the future. Not used any more

How to reproduce:

docker-compose up --build

then

docker exec -it <name_of_master_container> bash
root@master:/usr/spark-2.3.1# pyspark
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
2018-07-29 20:56:26 WARN  SparkConf:66 - The configuration key 'spark.executor.port' has been deprecated as of Spark 2.0.0 and may be removed in the future. Not used any more
2018-07-29 20:56:27 WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
2018-07-29 20:56:27 WARN  SparkConf:66 - The configuration key 'spark.executor.port' has been deprecated as of Spark 2.0.0 and may be removed in the future. Not used any more
2018-07-29 20:56:27 WARN  SparkConf:66 - The configuration key 'spark.executor.port' has been deprecated as of Spark 2.0.0 and may be removed in the future. Not used any more
2018-07-29 20:56:27 WARN  SparkConf:66 - The configuration key 'spark.executor.port' has been deprecated as of Spark 2.0.0 and may be removed in the future. Not used any more
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.3.1
      /_/

Using Python version 3.5.3 (default, Jan 19 2017 14:11:04)
SparkSession available as 'spark'.

Changes

removed deprecated 'spark.executor.port' from configuration

serge-m commented 6 years ago

@bryceageno

dylanmei commented 6 years ago

We can also remove these ports from the docker-compose.yml file as well.

serge-m commented 6 years ago

@dylanmei removed:

output after fix 
$ docker-compose up
master_1  | 2018-07-31 16:58:31 INFO  Master:2611 - Started daemon with process name: 1@master
master_1  | 2018-07-31 16:58:31 INFO  SignalUtils:54 - Registered signal handler for TERM
master_1  | 2018-07-31 16:58:31 INFO  SignalUtils:54 - Registered signal handler for HUP
master_1  | 2018-07-31 16:58:31 INFO  SignalUtils:54 - Registered signal handler for INT
master_1  | 2018-07-31 16:58:32 WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
worker_1  | 2018-07-31 16:58:32 INFO  Worker:2611 - Started daemon with process name: 1@worker
master_1  | 2018-07-31 16:58:32 INFO  SecurityManager:54 - Changing view acls to: root
master_1  | 2018-07-31 16:58:32 INFO  SecurityManager:54 - Changing modify acls to: root
master_1  | 2018-07-31 16:58:32 INFO  SecurityManager:54 - Changing view acls groups to: 
master_1  | 2018-07-31 16:58:32 INFO  SecurityManager:54 - Changing modify acls groups to: 
master_1  | 2018-07-31 16:58:32 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
worker_1  | 2018-07-31 16:58:32 INFO  SignalUtils:54 - Registered signal handler for TERM
worker_1  | 2018-07-31 16:58:32 INFO  SignalUtils:54 - Registered signal handler for HUP
worker_1  | 2018-07-31 16:58:32 INFO  SignalUtils:54 - Registered signal handler for INT
master_1  | 2018-07-31 16:58:32 INFO  Utils:54 - Successfully started service 'sparkMaster' on port 7077.
master_1  | 2018-07-31 16:58:32 INFO  Master:54 - Starting Spark master at spark://master:7077
master_1  | 2018-07-31 16:58:32 INFO  Master:54 - Running Spark version 2.3.1
master_1  | 2018-07-31 16:58:32 INFO  log:192 - Logging initialized @2491ms
master_1  | 2018-07-31 16:58:33 INFO  Server:346 - jetty-9.3.z-SNAPSHOT
worker_1  | 2018-07-31 16:58:33 WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
master_1  | 2018-07-31 16:58:33 INFO  Server:414 - Started @2683ms
master_1  | 2018-07-31 16:58:33 INFO  AbstractConnector:278 - Started ServerConnector@7e47b4dc{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
master_1  | 2018-07-31 16:58:33 INFO  Utils:54 - Successfully started service 'MasterUI' on port 8080.
master_1  | 2018-07-31 16:58:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2088e53f{/app,null,AVAILABLE,@Spark}
master_1  | 2018-07-31 16:58:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c3d81d8{/app/json,null,AVAILABLE,@Spark}
master_1  | 2018-07-31 16:58:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@32cd11b4{/,null,AVAILABLE,@Spark}
master_1  | 2018-07-31 16:58:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1d03277f{/json,null,AVAILABLE,@Spark}
master_1  | 2018-07-31 16:58:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7dd05646{/static,null,AVAILABLE,@Spark}
master_1  | 2018-07-31 16:58:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2c07168a{/app/kill,null,AVAILABLE,@Spark}
master_1  | 2018-07-31 16:58:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3e5e4de1{/driver/kill,null,AVAILABLE,@Spark}
master_1  | 2018-07-31 16:58:33 INFO  MasterWebUI:54 - Bound MasterWebUI to 0.0.0.0, and started at http://localhost:8080
master_1  | 2018-07-31 16:58:33 INFO  Server:346 - jetty-9.3.z-SNAPSHOT
master_1  | 2018-07-31 16:58:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@cf749f9{/,null,AVAILABLE}
master_1  | 2018-07-31 16:58:33 INFO  AbstractConnector:278 - Started ServerConnector@7a4078e4{HTTP/1.1,[http/1.1]}{master:6066}
master_1  | 2018-07-31 16:58:33 INFO  Server:414 - Started @2954ms
worker_1  | 2018-07-31 16:58:33 INFO  SecurityManager:54 - Changing view acls to: root
master_1  | 2018-07-31 16:58:33 INFO  Utils:54 - Successfully started service on port 6066.
master_1  | 2018-07-31 16:58:33 INFO  StandaloneRestServer:54 - Started REST server for submitting applications on port 6066
worker_1  | 2018-07-31 16:58:33 INFO  SecurityManager:54 - Changing modify acls to: root
worker_1  | 2018-07-31 16:58:33 INFO  SecurityManager:54 - Changing view acls groups to: 
worker_1  | 2018-07-31 16:58:33 INFO  SecurityManager:54 - Changing modify acls groups to: 
worker_1  | 2018-07-31 16:58:33 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
master_1  | 2018-07-31 16:58:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@a9bdded{/metrics/master/json,null,AVAILABLE,@Spark}
master_1  | 2018-07-31 16:58:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4968513c{/metrics/applications/json,null,AVAILABLE,@Spark}
worker_1  | 2018-07-31 16:58:33 INFO  Utils:54 - Successfully started service 'sparkWorker' on port 8881.
master_1  | 2018-07-31 16:58:33 INFO  Master:54 - I have been elected leader! New state: ALIVE
worker_1  | 2018-07-31 16:58:34 INFO  Worker:54 - Starting Spark worker 172.17.0.4:8881 with 2 cores, 1024.0 MB RAM
worker_1  | 2018-07-31 16:58:34 INFO  Worker:54 - Running Spark version 2.3.1
worker_1  | 2018-07-31 16:58:34 INFO  Worker:54 - Spark home: /usr/spark-2.3.1
worker_1  | 2018-07-31 16:58:34 INFO  log:192 - Logging initialized @3034ms
worker_1  | 2018-07-31 16:58:34 INFO  Server:346 - jetty-9.3.z-SNAPSHOT
worker_1  | 2018-07-31 16:58:34 INFO  Server:414 - Started @3137ms
worker_1  | 2018-07-31 16:58:34 INFO  AbstractConnector:278 - Started ServerConnector@63a57dc7{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
worker_1  | 2018-07-31 16:58:34 INFO  Utils:54 - Successfully started service 'WorkerUI' on port 8081.
worker_1  | 2018-07-31 16:58:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5bbddf26{/logPage,null,AVAILABLE,@Spark}
worker_1  | 2018-07-31 16:58:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4644c7df{/logPage/json,null,AVAILABLE,@Spark}
worker_1  | 2018-07-31 16:58:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4aff2dc{/,null,AVAILABLE,@Spark}
worker_1  | 2018-07-31 16:58:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4f6af41f{/json,null,AVAILABLE,@Spark}
worker_1  | 2018-07-31 16:58:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2ff0a812{/static,null,AVAILABLE,@Spark}
worker_1  | 2018-07-31 16:58:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@29f605c9{/log,null,AVAILABLE,@Spark}
worker_1  | 2018-07-31 16:58:34 INFO  WorkerWebUI:54 - Bound WorkerWebUI to 0.0.0.0, and started at http://localhost:8081
worker_1  | 2018-07-31 16:58:34 INFO  Worker:54 - Connecting to master master:7077...
worker_1  | 2018-07-31 16:58:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3cd61f42{/metrics/json,null,AVAILABLE,@Spark}
worker_1  | 2018-07-31 16:58:34 INFO  TransportClientFactory:267 - Successfully created connection to master/172.17.0.3:7077 after 55 ms (0 ms spent in bootstraps)
master_1  | 2018-07-31 16:58:34 INFO  Master:54 - Registering worker 172.17.0.4:8881 with 2 cores, 1024.0 MB RAM
worker_1  | 2018-07-31 16:58:34 INFO  Worker:54 - Successfully registered with master spark://master:7077
master_1  | 2018-07-31 16:58:50 INFO  Master:54 - Registering app PySparkShell
master_1  | 2018-07-31 16:58:50 INFO  Master:54 - Registered app PySparkShell with ID app-20180731165850-0000
.....