KylinOLAP / Kylin

This code base is retained for historical interest only, please visit Apache Incubator Repo for latest one
https://github.com/apache/incubator-kylin
Apache License 2.0
562 stars 225 forks source link

Facing issue while building Kylin Cube need help to fix it Cube is not building, logs are given below #509

Open fakhar101 opened 2 years ago

fakhar101 commented 2 years ago

java.io.IOException: OS command error exit with return code: 1, error message: log4j: Using URL [file:/home/hdoop/apache-kylin-4.0.1-bin-spark3/conf/spark-driver-log4j.properties] for automatic log4j configuration. log4j: Reading configuration from URL file:/home/hdoop/apache-kylin-4.0.1-bin-spark3/conf/spark-driver-log4j.properties log4j: Parsing for [root] with value=[INFO,hdfs]. log4j: Level token is [INFO]. log4j: Category root set to INFO log4j: Parsing appender named "hdfs". log4j: Parsing layout options for "hdfs". log4j: Setting property [conversionPattern] to [%d{ISO8601} %-5p [%t] %c{2} : %m%n]. log4j: End of parsing for "hdfs". log4j: Setting property [hdfsWorkingDir] to [s3a://bucketpath]. log4j: Setting property [kerberosPrincipal] to []. log4j: Setting property [logPath] to [s3a://bucketpath/execute_output.json.1654872379334.log]. log4j: Setting property [kerberosEnable] to [false]. log4j: Setting property [kerberosKeytab] to []. log4j: Setting property [logQueueCapacity] to [5000]. log4j: Setting property [flushInterval] to [5000]. log4j:WARN SparkDriverHdfsLogAppender starting ... log4j:WARN hdfsWorkingDir -> s3a://devopscmd-prod/apachekylin/kylin_metadata/ log4j:WARN spark.driver.log4j.appender.hdfs.File -> s3a://bucketpath/execute_output.json.1654872379334.log log4j:WARN kerberosEnable -> false log4j:WARN SparkDriverHdfsLogAppender started ... log4j: Parsed "hdfs" options. log4j: Parsing for [org.springframework] with value=[WARN]. log4j: Level token is [WARN]. log4j: Category org.springframework set to WARN log4j: Handling log4j.additivity.org.springframework=[null] log4j: Parsing for [org.apache.spark] with value=[WARN]. log4j: Level token is [WARN]. log4j: Category org.apache.spark set to WARN log4j: Handling log4j.additivity.org.apache.spark=[null] log4j: Parsing for [org.apache.kylin] with value=[DEBUG]. log4j: Level token is [DEBUG]. log4j: Category org.apache.kylin set to DEBUG log4j: Handling log4j.additivity.org.apache.kylin=[null] log4j: Finished configuring. log4j:WARN SparkDriverHdfsLogAppender flush log when shutdown ... The command is: export HADOOP_CONF_DIR=/home/hdoop/apache-kylin-4.0.1-bin-spark3/hadoop_conf && /home/hdoop/spark-3.1.1-bin-hadoop3.2/bin/spark-submit --class org.apache.kylin.engine.spark.application.SparkEntry --conf 'spark.yarn.queue=default' --conf 'spark.history.fs.logDirectory=s3a://bucket/apachekylin/' --conf 'spark.driver.extraJavaOptions=-XX:+CrashOnOutOfMemoryError -Dlog4j.configuration=file:/home/hdoop/apache-kylin-4.0.1-bin-spark3/conf/spark-driver-log4j.properties -Dkylin.kerberos.enabled=false -Dkylin.hdfs.working.dir=s3a://devopscmd-prod/apachekylin/kylin_metadata/ -Dspark.driver.log4j.appender.hdfs.File=s3a://bucketpath/execute_output.json.1654872379334.log -Dlog4j.debug=true -Dspark.driver.rest.server.address=ipaddress:7070 -Dspark.driver.param.taskId=ddf325d9-5232-422e-a5ff-0f35e4b0949c-00 -Dspark.driver.local.logDir=/home/hdoop/apache-kylin-4.0.1-bin-spark3/logs/spark' --conf 'spark.master=local' --conf 'spark.hadoop.yarn.timeline-service.enabled=false' --conf 'spark.driver.cores=1' --conf 'spark.eventLog.enabled=true' --conf 'spark.eventLog.dir=s3a://bucket/apachekylin/' --conf 'spark.driver.memory=2G' --conf 'spark.driver.memoryOverhead=512M' --conf 'spark.sql.autoBroadcastJoinThreshold=-1' --conf 'spark.sql.adaptive.enabled=false' --conf 'spark.driver.extraClassPath=/home/hdoop/apache-kylin-4.0.1-bin-spark3/lib/kylin-parquet-job-4.0.1.jar' --name job_step_ddf325d9-5232-422e-a5ff-0f35e4b0949c-00 --jars /home/hdoop/apache-kylin-4.0.1-bin-spark3/lib/kylin-parquet-job-4.0.1.jar /home/hdoop/apache-kylin-4.0.1-bin-spark3/lib/kylin-parquet-job-4.0.1.jar -className org.apache.kylin.engine.spark.job.ResourceDetectBeforeCubingJob s3a://bucketpath/ddf325d9-5232-422e-a5ff-0f35e4b0949c-00_jobId at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:98) at org.apache.kylin.engine.spark.job.NSparkExecutable.runSparkSubmit(NSparkExecutable.java:282) at org.apache.kylin.engine.spark.job.NSparkExecutable.doWork(NSparkExecutable.java:168) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:206) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:94) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:206) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``