DTStack / Taier

Taier is a big data development platform for submission, scheduling, operation and maintenance, and indicator information display
https://dtstack.github.io/Taier/
Apache License 2.0
1.33k stars 334 forks source link

[Question] Spark SQL run on HDP bad substitution #1102

Closed vainhope closed 1 year ago

vainhope commented 1 year ago

Search before asking

Description

HDP run spark sql error messge:

/data/1/hadoop/yarn/local/usercache/scuser/appcache/application_1689728731929_2027/container_e8579_1689728731929_2027_01_000004/launch_container.sh: line 33: $PWD:$PWD/spark_conf:$PWD/spark_libs/:$HADOOP_CONF_DIR:/usr/hdp/3.0.0.0-1634/hadoop/:/usr/hdp/3.0.0.0-1634/hadoop/lib/:/usr/hdp/current/hadoop-hdfs-client/:/usr/hdp/current/hadoop-hdfs-client/lib/:/usr/hdp/current/hadoop-yarn-client/:/usr/hdp/current/hadoop-yarn-client/lib/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/:$PWD/mr-framework/hadoop/share/hadoop/common/:$PWD/mr-framework/hadoop/share/hadoop/common/lib/:$PWD/mr-framework/hadoop/share/hadoop/yarn/:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/:$PWD/mr-framework/hadoop/share/hadoop/hdfs/:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure:$PWD/spark_conf/hadoop_conf: bad substitution End of LogType:prelaunch.err

Code of Conduct

vainhope commented 1 year ago

环境参数手动新增

spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0-2041 (your installed HDP version)

spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0-2041 (your installed HDP version)