search

gettyimages / docker-spark

Docker build for Apache Spark
MIT License
679 stars 369 forks source link

how can I up a worker use mesos uri? #31

Open Senmumu opened 6 years ago

Senmumu commented 6 years ago

I want to up a worker and its master will be mesos-master which I have set up. however ,when I run docker-compose up ,the console shows this


worker_1  | Exception in thread "main" org.apache.spark.SparkException: Invalid master URL: spark://mesos://zk://zk1:2181 ```

this is docker-compose.yml

```worker:
  image: gettyimages/spark:2.2.0-hadoop-2.7
  command: bin/spark-class org.apache.spark.deploy.worker.Worker  mesos://zk://zk1:2181,zk2:2181,zk3:2181/mesos
  hostname: worker
  environment:
    SPARK_CONF_DIR: /conf
    SPARK_WORKER_CORES: 2
    SPARK_WORKER_MEMORY: 1g
    SPARK_WORKER_PORT: 8881
    SPARK_WORKER_WEBUI_PORT: 8081
    SPARK_PUBLIC_DNS: localhost
  expose:
    - 7012
    - 7013
    - 7014
    - 7015
    - 7016
    - 8881
  ports:
    - 8081:8081
  volumes:
    - ./conf/worker:/conf
    - ./data:/tmp/data```