search

mmolimar / ksql-jdbc-driver

JDBC driver for Apache Kafka
Apache License 2.0
88 stars 19 forks source link

Error with STRUCT Dereference operator #16

Closed michaelper22 closed 5 years ago

michaelper22 commented 5 years ago

Hello,

My team is running into an issue with the STRUCT Dereference operator. We have the following stream in KSQL (the orders_topic topic is produced by ksql-datagen):

CREATE STREAM orders_raw (
        itemid VARCHAR,
        orderunits DOUBLE,
        address STRUCT<
            city VARCHAR,
            state VARCHAR,
            zipcode INT>,
        ordertime VARCHAR)
     WITH (
        KAFKA_TOPIC='orders_topic',
        VALUE_FORMAT='JSON');

From the ksql CLI, this works as expected:

ksql> select address->city from orders_raw limit 5;
City_82
City_34
City_72
City_98
City_46
Limit Reached
Query terminated

When we run this query through the JDBC driver, we get the following:

java.sql.SQLException: Error getting metadata for query: 'SELECT ADDRESS->CITY FROM ORDERS_RAW LIMIT 5;'. Error: io.confluent.ksql.analyzer.ExpressionAnalyzer$Visitor.visitDereferenceExpression(ExpressionAnalyzer.java:120)
io.confluent.ksql.parser.tree.DereferenceExpression.accept(DereferenceExpression.java:54)
io.confluent.ksql.parser.tree.AstVisitor.process(AstVisitor.java:22)
io.confluent.ksql.analyzer.ExpressionAnalyzer.analyzeExpression(ExpressionAnalyzer.java:47)
io.confluent.ksql.analyzer.Analyzer.analyzeExpressions(Analyzer.java:203)
io.confluent.ksql.analyzer.Analyzer.visitQuerySpecification(Analyzer.java:130)
io.confluent.ksql.analyzer.Analyzer.visitQuerySpecification(Analyzer.java:68)
io.confluent.ksql.parser.tree.QuerySpecification.accept(QuerySpecification.java:136)
io.confluent.ksql.parser.tree.AstVisitor.process(AstVisitor.java:22)
io.confluent.ksql.parser.DefaultTraversalVisitor.visitQuery(DefaultTraversalVisitor.java:126)
io.confluent.ksql.parser.tree.Query.accept(Query.java:64)
io.confluent.ksql.parser.tree.AstVisitor.process(AstVisitor.java:22)
io.confluent.ksql.analyzer.QueryAnalyzer.analyze(QueryAnalyzer.java:44)
io.confluent.ksql.QueryEngine.buildQueryLogicalPlan(QueryEngine.java:111)
io.confluent.ksql.QueryEngine.buildLogicalPlans(QueryEngine.java:86)
io.confluent.ksql.KsqlEngine.getQueryExecutionPlan(KsqlEngine.java:260)
io.confluent.ksql.rest.server.resources.KsqlResource.lambda$registerKsqlStatementTasks$21(KsqlResource.java:634)
io.confluent.ksql.rest.server.resources.KsqlResource.getStatementExecutionPlan(KsqlResource.java:619)
io.confluent.ksql.rest.server.resources.KsqlResource.explainQuery(KsqlResource.java:601)
...

Can you think of something that's wrong in our query, or a possible bug in KSQL or your driver?

Thanks, Michael

mmolimar commented 5 years ago

Hi @michaelper22

When executing a SQL query, the driver execute as well an EXPLAIN command to know which are the values to map (with the same query you executed). If you try this command, it shouldn't work. Can try to execute that query without the nested field? I mean, like this select address from orders_raw limit 5

michaelper22 commented 5 years ago

Yup, tried doing an EXPLAIN on the query from the KSQL console, and got the error "Column CITY cannot be resolved". Maybe we'll open a bug with them. Thank you for the suggestion.