For now, spark Livy status check could not detect statement output level error. That is to say, the SQL level exception is not captured. We need to update the code in HelpFun.getTaskStatusSpark to further check the statement output level status as well as the exception. Once the status is FAILED, so the error information in UI. Below is a sample response which is still shown FINISHED in web UI.
{
"id": 0,
"code": "a = sqlContext.sql(\"select refresh_time, symbol, ask_size from stock_source\").take(10)\n%table a",
"state": "available",
"output": {
"status": "error",
"execution_count": 0,
"ename": "AnalysisException",
"evalue": "u'Table or view not found: stock_source; line 1 pos 43'",
"traceback": [
"Traceback (most recent call last):\n",
" File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/context.py\", line 384, in sql\n return self.sparkSession.sql(sqlQuery)\n",
" File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py\", line 556, in sql\n return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)\n",
" File \"/opt/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py\", line 1133, in call\n answer, self.gateway_client, self.target_id, self.name)\n",
" File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 69, in deco\n raise AnalysisException(s.split(': ', 1)[1], stackTrace)\n",
"AnalysisException: u'Table or view not found: stock_source; line 1 pos 43'\n"
]
},
"progress": 1
}
datafibers
commented
6 years ago
For now, spark Livy status check could not detect statement output level error. That is to say, the SQL level exception is not captured. We need to update the code in HelpFun.getTaskStatusSpark to further check the statement output level status as well as the exception. Once the status is FAILED, so the error information in UI. Below is a sample response which is still shown FINISHED in web UI.
{ "id": 0, "code": "a = sqlContext.sql(\"select refresh_time, symbol, ask_size from stock_source\").take(10)\n%table a", "state": "available", "output": { "status": "error", "execution_count": 0, "ename": "AnalysisException", "evalue": "u'Table or view not found: stock_source; line 1 pos 43'", "traceback": [ "Traceback (most recent call last):\n", " File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/context.py\", line 384, in sql\n return self.sparkSession.sql(sqlQuery)\n", " File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/session.py\", line 556, in sql\n return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)\n", " File \"/opt/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py\", line 1133, in call\n answer, self.gateway_client, self.target_id, self.name)\n", " File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 69, in deco\n raise AnalysisException(s.split(': ', 1)[1], stackTrace)\n", "AnalysisException: u'Table or view not found: stock_source; line 1 pos 43'\n" ] }, "progress": 1 }