We know the property logDetail uses nested type. XSQL parse nested type of ElasticSearch as nested. The nested just used for display. For example:
spark-xsql> desc details;
20/01/22 14:49:16 INFO SparkXSQLShell: current SQL: desc details
20/01/22 14:49:16 WARN SparkXSQLShell: hive.cli.print.header not configured, so doesn't print colum's name.
hasLog boolean NULL
logDetail nested NULL
md5 keyword NULL
sha1 keyword NULL
sha256 keyword NULL
Time taken: 0.269 s
The type of column logDetail display as nested. But Spark can't use this type, we need to transform the nested type of ElasticSearch as StructType or ArrayType(StructType).
Then we can query data with the sub properties of nested type, such as:
select hasLog, logDetail.confidence from details limit 5;
select logDetail.confidence from details group by logDetail.confidence;
select * from details where logDetail.confidence[0]="HEURISTIC";
This PR will resolve the issue and parse nested as the data type that Spark can use.
spark-xsql> desc details;
20/01/22 15:07:09 INFO SparkXSQLShell: current SQL: desc details
20/01/22 15:07:13 WARN SparkXSQLShell: hive.cli.print.header not configured, so doesn't print colum's name.
hasLog boolean NULL
logDetail array<struct<confidence:string,danger:string,dbt:string,detectNotify:boolean,virusName:string>> NULL
md5 string NULL
sha1 string NULL
sha256 string NULL
Time taken: 3.768 s
What changes were proposed in this pull request?
The ElasticSearch index log contains a type details. The mapping of details is:
We know the property
logDetail
uses nested type.XSQL
parse nested type of ElasticSearch asnested
. Thenested
just used for display. For example:The type of column logDetail display as
nested
. But Spark can't use this type, we need to transform the nested type of ElasticSearch as StructType or ArrayType(StructType). Then we can query data with the sub properties of nested type, such as:This PR will resolve the issue and parse nested as the data type that Spark can use.
Note: This PR reference https://github.com/apache/spark/pull/23353
How was this patch tested?
No UT.