search

cloudera-labs / envelope

Build configuration-driven ETL pipelines on Apache Spark
Apache License 2.0
158 stars 89 forks source link

Envelope CAN NOT run YARN #32

Closed 383280199 closed 5 years ago

383280199 commented 5 years ago

spark-submit2 envelope.jar XXX.conf it cannot run YARN，because the spark-related configuration is in the "application" of the xxx.conf file, the sparksubmit cannot be obtained at startup. In other words, the driver will start locally instead of a node in the cluster. so if we want to run cluster throgh then YARN , We need to extract the “application" from the xxx.conf file, get a separate configuration key-value configuration file, and then add the proposed file after the spark-submit2 --conf. e.g. spark-submit2 --conf application.conf envelope.jar xxx.conf

The above is my suggestion. I don't know if I have misinterpreted the program. Please advise.

jeremybeard commented 5 years ago

You can add configuration to the spark2-submit call, so if you want to run in YARN cluster mode you can add --deploy-mode cluster instead of trying to put it in the configuration file.

For future reference we also have the Cloudera Community forum for questions like these.