Issue: Running spark-connect and run.sh script together in Docker Compose

parbatrajpurohit commented 1 month ago

Name and Version

bitnami/spark:3.5.1

What is the problem this feature will solve?

Description: I am trying to set up a Spark cluster using Docker Compose with the following services:

spark-master spark-connect spark-worker spark-worker2

The Docker Compose file I am using is as follows:

version: '3.8'
services:
  spark-master:
    image: bitnami/spark
    container_name: spark-master
    environment:
      - SPARK_MODE=master
      - SPARK_MASTER_WEBUI_PORT=8080
      - SPARK_MASTER_PORT=7077
      - SPARK_SUBMIT_OPTIONS=--packages io.delta:delta-spark_2.12:3.2.0
      - SPARK_MASTER_HOST=spark-master
    ports:
      - 8080:8080
      - 7077:7077
    networks:
      - spark-network
    volumes:
      - /mnt/f/My_docs/Project/spark:/mnt

  spark-connect:
    image: bitnami/spark
    container_name: spark-connect
    environment:
      - SPARK_MODE=master
      - SPARK_MASTER=spark://spark-master:7077
    ports:
      - 15002:15002
    networks:
      - spark-network
    depends_on:
      - spark-master
    command: ["/bin/bash", "-c", "/opt/bitnami/spark/sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.5.1"]
    volumes:
      - /mnt/f/My_Docs/Project/spark:/mnt

  spark-worker:
    image: bitnami/spark
    container_name: spark-worker
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_WORKER_CORES=2
      - SPARK_WORKER_MEMORY=2G
      - SPARK_WORKER_WEBUI_PORT=8081
    ports:
      - 8081:8081
    depends_on:
      - spark-master
    networks:
      - spark-network

  spark-worker2:
    image: bitnami/spark
    container_name: spark-worker2
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER=spark://spark-master:7077
      - SPARK_WORKER_CORES=2
      - SPARK_WORKER_MEMORY=2G
      - SPARK_WORKER_WEBUI_PORT=8082
    ports:
      - 8082:8082
    depends_on:
      - spark-master
    networks:
      - spark-network
networks:
  spark-network:

What is the feature you are proposing to solve the problem?

I need to run a custom script (run.sh) along with the start-connect-server.sh service. However, I am unsure how to modify the command section in the spark-connect service to achieve this.

Expected Solution:

Instructions or an example of how to modify the command section to run both the spark-connect server and the run.sh script.
Any required changes to the Docker Compose file to accommodate this setup.

carrodher commented 1 month ago

The issue may not be directly related to the Bitnami container image/Helm chart, but rather to how the application is being utilized, configured in your specific environment, or tied to a specific scenario that is not easy to reproduce on our side.

If you think that's not the case and are interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

Suppose you have any questions about the application, customizing its content, or technology and infrastructure usage. In that case, we highly recommend that you refer to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

github-actions[bot] commented 3 weeks ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 2 weeks ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

bitnami / containers