search

yTek01 / docker-spark-airflow

30 stars 16 forks source link

Cant run spark_airflow_dag on Airflow #2

Open mukul-B opened 1 year ago

mukul-B commented 1 year ago

i am able to run "docker exec -it docker-spark-airflow-spark-worker-1 spark-submit --master spark://bd1c9dde9216:7077 spark_etl_script_docker.py" image but when i trigger "Spark_submit_task" it is giving error

image

fiatttkk commented 1 year ago

Anyone solved it?

fiatttkk commented 1 year ago

i am able to run "docker exec -it docker-spark-airflow-spark-worker-1 spark-submit --master spark://bd1c9dde9216:7077 spark_etl_script_docker.py" image but when i trigger "Spark_submit_task" it is giving error

image

Anyone solved it ?

sl2902 commented 3 months ago

It looks like it is running yarn instead of in client mode. Check the connection settings