search

databrickslabs / transpiler

SIEM-to-Spark Transpiler
Other
42 stars 5 forks source link

NoSuchMethodErrors on cidr_match queries #115

Open mstair opened 1 year ago

mstair commented 1 year ago

Getting following error on attempts to run cidr_match queries against following CREATE TABLE security_log (user_id STRING, remote_host_ip STRING). Cluster is 11.3 LTS ML (includes Apache Spark 3.3.0, Scala 2.12) 

from dbl_transpiler import spl

display(spl("""
index=security_log remote_host_ip=109.177.0.0/16
"""))

...

Py4JJavaError: An error occurred while calling z:com.databricks.labs.transpiler.spl.Transpiler.toDataFrame.
: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.expressions.Cast$.apply$default$4()Z
    at org.apache.spark.sql.CidrMatch.aton(CidrMatch.scala:43)
    at org.apache.spark.sql.CidrMatch.ipAddress(CidrMatch.scala:24)
    at org.apache.spark.sql.CidrMatch.cidrMatch(CidrMatch.scala:22)
mstair commented 1 year ago

For completeness, I tried to run on DB 12.0/12.2/13.0 clusters and got the same error.

mstair commented 1 year ago

Same error on 13.2 cluster also. @nfx any update?