search

apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
https://linkis.apache.org/
Apache License 2.0
3.32k stars 1.17k forks source link

Qualitis执行规则时报错Execute code error for org.apache.spark.sql.AnalysisException: Table or view not found[Question] #5092

Closed penny321 closed 8 months ago

penny321 commented 8 months ago

Before asking

Your environment

Describe your questions

Q1. linkis环境单独执行spark引擎任务正常，在qualitis执行规则时，报错Execute code error for org.apache.spark.sql.AnalysisException: Table or view not found

Eureka service list

eg: eureka

Some logs info or acctch file

linkis-xxx.log:



<!--日志文字贴到这里-->
2024-03-13 11:16:19.016 INFO Program is substituting variables for you
2024-03-13 11:16:19.016 INFO Variables substitution ended successfully
Job with jobId : IDE_huilan_spark_0 and execID : IDE_huilan_spark_0 submitted 
2024-03-13 11:16:19.016 INFO You have submitted a new job, script code (after variable substitution) is
************************************SCRIPT CODE************************************
import java.sql.{Connection, DriverManager}
val prop = new java.util.Properties;
prop.setProperty("user", "******");
prop.setProperty("password", "******");
val UUID = java.util.UUID.randomUUID.toString
val tmp1 = spark.sql("select * from media_db.t_report where (1=1) and (target_name is null)");
val schemas = tmp1.schema.fields.map(f => f.name).toList
val newSchemas = schemas.map(s => s.replaceAll("[()]", "")).toList
val tmp2 = tmp1.toDF(newSchemas: _*)
spark.sqlContext.setConf("hive.exec.dynamic.partition", "true")
spark.sqlContext.setConf("hive.exec.dynamic.partition.mode", "nonstrict")
spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")
if (spark.catalog.tableExists("huilan_ind.check_test_test_null")) {
tmp2.withColumn("qualitis_partition_key", lit("20240313")).write.mode("overwrite").insertInto("huilan_ind.check_test_test_null");
} else {
tmp2.withColumn("qualitis_partition_key", lit("20240313")).write.mode("append").partitionBy("qualitis_partition_key").format("hive").saveAsTable("huilan_ind.check_test_test_null");
}
tmp2.selectExpr("count(*) as value", "'QUALITIS20240313111618596_621728' as application_id", "'Long' as result_type", "'7' as rule_id", "'-1' as rule_metric_id", "'-1' as run_date", "'2024-03-13 11:16:18' as create_time").write.mode(org.apache.spark.sql.SaveMode.Append).jdbc("jdbc:mysql://172.20.0.71:3306/qualitis?useSSL=false&createDatabaseIfNotExist=true&useUnicode=true&characterEncoding=utf-8", "qualitis_application_task_result", prop)
************************************SCRIPT CODE************************************
2024-03-13 11:16:19.016 INFO Your job is accepted,  jobID is IDE_huilan_spark_0 and taskID is 13 in ServiceInstance(linkis-cg-entrance, huilan71:6104). Please wait it to be scheduled
job is scheduled.
2024-03-13 11:16:19.016 INFO Your job is Scheduled. Please wait it to run.
Your job is being scheduled by orchestrator.
2024-03-13 11:16:19.016 INFO job is running.
2024-03-13 11:16:19.016 INFO Your job is Running now. Please wait it to complete.
2024-03-13 11:16:19.016 INFO Job with jobGroupId : 13 and subJobId : 11 was submitted to Orchestrator.
2024-03-13 11:16:19.016 INFO Background is starting a new engine for you,execId astJob_1_codeExec_1 mark id is mark_1, it may take several seconds, please wait
2024-03-13 11:17:29.017 INFO EngineConn local log path: ServiceInstance(linkis-cg-engineconn, huilan71:37767) /home/huilan/linkis/back/appcom/tmp/huilan/workDir/c1341aca-1c90-44d6-b492-b49581f51f18/logs
2024-03-13 11:17:29.017 INFO yarn application id: application_1710127870671_0002
scala> import java.sql.{Connection, DriverManager}
2024-03-13 11:17:47.017 INFO yarn application id: application_1710127870671_0002
scala> val prop = new java.util.Properties;
2024-03-13 11:17:48.017 INFO yarn application id: application_1710127870671_0002
scala> prop.setProperty("user", "******");
2024-03-13 11:17:48.017 INFO yarn application id: application_1710127870671_0002
scala> prop.setProperty("password", "******");
2024-03-13 11:17:48.017 INFO yarn application id: application_1710127870671_0002
scala> val UUID = java.util.UUID.randomUUID.toString
2024-03-13 11:17:49.017 INFO yarn application id: application_1710127870671_0002
2024-03-13 11:17:29.071 WARN  [Linkis-Default-Scheduler-Thread-3] org.apache.linkis.engineconn.computation.executor.hook.executor.ExecuteOnceHook 50 warn - execute once become effective, register lock listener
2024-03-13 11:17:29.164 WARN  [Linkis-Default-Scheduler-Thread-3] org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor 50 warn - Start to init sparkILoop cost 2.
2024-03-13 11:17:47.246 WARN  [Linkis-Default-Scheduler-Thread-3] org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor 50 warn - Finished to init sparkILoop cost 18083.
2024-03-13 11:18:06.520 ERROR [Linkis-Default-Scheduler-Thread-3] org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor 62 error - Execute code error for org.apache.spark.sql.AnalysisException: Table or view not found: `media_db`.`t_report`; line 1 pos 14;
'Project [*]
+- 'Filter ((1 = 1) && isnull('target_name))
   +- 'UnresolvedRelation `media_db`.`t_report`
  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:90)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:85)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:85)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
  at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
  at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
  at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:78)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
  ... 82 elided
2024-03-13 11:18:06.524 ERROR [Linkis-Default-Scheduler-Thread-3] org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor 58 error - execute code failed! org.apache.linkis.engineplugin.spark.exception.ExecuteError: errCode: 40005 ,desc: execute sparkScala failed! ,ip: huilan71 ,port: 37767 ,serviceKind: linkis-cg-engineconn
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor$$anonfun$1.apply(SparkScalaExecutor.scala:192) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor$$anonfun$1.apply(SparkScalaExecutor.scala:156) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) ~[scala-library-2.11.12.jar:?]
    at scala.Console$.withOut(Console.scala:65) ~[scala-library-2.11.12.jar:?]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.executeLine(SparkScalaExecutor.scala:155) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor$$anonfun$runCode$1.apply$mcV$sp(SparkScalaExecutor.scala:130) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor$$anonfun$runCode$1.apply(SparkScalaExecutor.scala:130) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor$$anonfun$runCode$1.apply(SparkScalaExecutor.scala:130) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) ~[linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.runCode(SparkScalaExecutor.scala:131) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor$$anonfun$executeLine$2$$anonfun$2.apply(SparkEngineConnExecutor.scala:83) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor$$anonfun$executeLine$2$$anonfun$2.apply(SparkEngineConnExecutor.scala:83) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor$$anonfun$executeLine$2.apply(SparkEngineConnExecutor.scala:83) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor$$anonfun$executeLine$2.apply(SparkEngineConnExecutor.scala:64) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor.executeLine(SparkEngineConnExecutor.scala:91) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10$$anonfun$apply$11.apply(ComputationExecutor.scala:180) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10$$anonfun$apply$11.apply(ComputationExecutor.scala:179) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) ~[linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10.apply(ComputationExecutor.scala:181) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10.apply(ComputationExecutor.scala:175) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at scala.collection.immutable.Range.foreach(Range.scala:160) [scala-library-2.11.12.jar:?]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2.apply(ComputationExecutor.scala:174) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2.apply(ComputationExecutor.scala:150) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) [linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.toExecuteTask(ComputationExecutor.scala:227) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$3.apply(ComputationExecutor.scala:242) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$3.apply(ComputationExecutor.scala:242) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) [linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.acessible.executor.entity.AccessibleExecutor.ensureIdle(AccessibleExecutor.scala:55) [linkis-accessible-executor-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.acessible.executor.entity.AccessibleExecutor.ensureIdle(AccessibleExecutor.scala:49) [linkis-accessible-executor-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.ensureOp(ComputationExecutor.scala:134) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.execute(ComputationExecutor.scala:241) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl.org$apache$linkis$engineconn$computation$executor$service$TaskExecutionServiceImpl$$executeTask(TaskExecutionServiceImpl.scala:240) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1$$anonfun$run$1.apply$mcV$sp(TaskExecutionServiceImpl.scala:173) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1$$anonfun$run$1.apply(TaskExecutionServiceImpl.scala:171) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1$$anonfun$run$1.apply(TaskExecutionServiceImpl.scala:171) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) [linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryAndWarn(Utils.scala:69) [linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1.run(TaskExecutionServiceImpl.scala:171) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_271]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_271]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_271]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_271]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_271]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_271]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_271]
2024-03-13 11:18:06.585 ERROR [Linkis-Default-Scheduler-Thread-3] org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl 58 error - org.apache.spark.sql.AnalysisException: Table or view not found: `media_db`.`t_report`; line 1 pos 14;
'Project [*]
+- 'Filter ((1 = 1) && isnull('target_name))
   +- 'UnresolvedRelation `media_db`.`t_report`
  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:90)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:85)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:85)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
  at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
  at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
  at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:78)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
  ... 82 elided
org.apache.linkis.engineplugin.spark.exception.ExecuteError: errCode: 40005 ,desc: execute sparkScala failed! ,ip: huilan71 ,port: 37767 ,serviceKind: linkis-cg-engineconn
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor$$anonfun$1.apply(SparkScalaExecutor.scala:192) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor$$anonfun$1.apply(SparkScalaExecutor.scala:156) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) ~[scala-library-2.11.12.jar:?]
    at scala.Console$.withOut(Console.scala:65) ~[scala-library-2.11.12.jar:?]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.executeLine(SparkScalaExecutor.scala:155) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor$$anonfun$runCode$1.apply$mcV$sp(SparkScalaExecutor.scala:130) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor$$anonfun$runCode$1.apply(SparkScalaExecutor.scala:130) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor$$anonfun$runCode$1.apply(SparkScalaExecutor.scala:130) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) ~[linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkScalaExecutor.runCode(SparkScalaExecutor.scala:131) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor$$anonfun$executeLine$2$$anonfun$2.apply(SparkEngineConnExecutor.scala:83) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor$$anonfun$executeLine$2$$anonfun$2.apply(SparkEngineConnExecutor.scala:83) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor$$anonfun$executeLine$2.apply(SparkEngineConnExecutor.scala:83) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor$$anonfun$executeLine$2.apply(SparkEngineConnExecutor.scala:64) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineplugin.spark.executor.SparkEngineConnExecutor.executeLine(SparkEngineConnExecutor.scala:91) ~[linkis-engineplugin-spark-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10$$anonfun$apply$11.apply(ComputationExecutor.scala:180) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10$$anonfun$apply$11.apply(ComputationExecutor.scala:179) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) ~[linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10.apply(ComputationExecutor.scala:181) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2$$anonfun$apply$10.apply(ComputationExecutor.scala:175) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at scala.collection.immutable.Range.foreach(Range.scala:160) ~[scala-library-2.11.12.jar:?]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2.apply(ComputationExecutor.scala:174) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$toExecuteTask$2.apply(ComputationExecutor.scala:150) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.toExecuteTask(ComputationExecutor.scala:227) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$3.apply(ComputationExecutor.scala:242) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor$$anonfun$3.apply(ComputationExecutor.scala:242) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryFinally(Utils.scala:61) ~[linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.acessible.executor.entity.AccessibleExecutor.ensureIdle(AccessibleExecutor.scala:55) ~[linkis-accessible-executor-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.acessible.executor.entity.AccessibleExecutor.ensureIdle(AccessibleExecutor.scala:49) ~[linkis-accessible-executor-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.ensureOp(ComputationExecutor.scala:134) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.execute.ComputationExecutor.execute(ComputationExecutor.scala:241) ~[linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl.org$apache$linkis$engineconn$computation$executor$service$TaskExecutionServiceImpl$$executeTask(TaskExecutionServiceImpl.scala:240) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1$$anonfun$run$1.apply$mcV$sp(TaskExecutionServiceImpl.scala:173) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1$$anonfun$run$1.apply(TaskExecutionServiceImpl.scala:171) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1$$anonfun$run$1.apply(TaskExecutionServiceImpl.scala:171) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryCatch(Utils.scala:40) [linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.common.utils.Utils$.tryAndWarn(Utils.scala:69) [linkis-common-1.0.3.jar:1.0.3]
    at org.apache.linkis.engineconn.computation.executor.service.TaskExecutionServiceImpl$$anon$1.run(TaskExecutionServiceImpl.scala:171) [linkis-computation-engineconn-1.0.3.jar:1.0.3]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_271]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_271]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_271]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_271]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_271]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_271]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_271]
Job with execId-IDE_huilan_spark_0 and subJobId : 11 from orchestrator completed with state ErrorExecuteResponse(21304, Task is Failed,errorMsg: org.apache.spark.sql.AnalysisException: Table or view not found: `media_db`.`t_report`; line 1 pos 14;
'Project [*]
+- 'Filter ((1 = 1) && isnull('target_name))
   +- 'UnresolvedRelation `media_db`.`t_report`
  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:90)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:85)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:85)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
  at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:57)
  at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:55)
  at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:47)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:78)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
  ... 82 elided
,null)
2024-03-13 11:18:07.018 INFO job is completed.
2024-03-13 11:18:07.018 INFO Task creation time(任务创建时间): 2024-03-13 11:16:19, Task scheduling time(任务调度时间): 2024-03-13 11:16:19, Task start time(任务开始时间): 2024-03-13 11:16:19, Mission end time(任务结束时间): 2024-03-13 11:18:07
2024-03-13 11:18:07.018 INFO Your mission(您的任务) 13 The total time spent is(总耗时时间为): 1.8 分钟
2024-03-13 11:18:07.018 INFO Sorry. Your job completed with a status Failed. You can view logs for the reason.

log file:
 <!-- 拖拽上传后，会自动生成如下示例格式的附件url -->
![qualitis](https://github.com/apache/linkis/assets/45955708/4c82145a-dec6-4ac8-bdb1-403b53b8e184)
github-actions[bot] commented 8 months ago

:blush: Welcome to the Apache Linkis community!!

We are glad that you are contributing by opening this issue.

Please make sure to include all the relevant context. We will be here shortly.

If you are interested in contributing to our website project, please let us know! You can check out our contributing guide on :point_right: How to Participate in Project Contribution.

Community

WeChat Assistant WeChat Public Account

Mailing Lists

Name Description Subscribe Unsubscribe Archive
dev@linkis.apache.org community activity information subscribe unsubscribe archive
peacewong commented 8 months ago

Is the hive-site configuration file not properly loaded by spark? 是不是hive-site配置文件没有正常被spark加载