search

knaufk / flink-faker

A data generator source connector for Flink SQL based on data-faker.
Apache License 2.0
211 stars 57 forks source link

Cannot use #{address.city} in expression #38

Open tsreaper opened 2 years ago

tsreaper commented 2 years ago

Run the following SQL in SQL client:

create temporary table faker ( a string ) with ( 'connector' = 'faker', 'number-of-rows' = '10', 'fields.a.expression' = '#{address.city}' );
select * from faker;

Exception:

Caused by: org.apache.flink.table.api.ValidationException: Unable to create a source for reading table 'default_catalog.default_database.faker'.

Table options are:

'connector'='faker'
'fields.a.expression'='#{address.city}'
'number-of-rows'='10'
    at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:159) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:184) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.createDynamicTableSource(CatalogSourceTable.java:178) ~[?:?]
    at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:116) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3585) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2507) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2144) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2093) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3438) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570) ~[?:?]
    at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:198) ~[?:?]
    at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:190) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:1240) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:1188) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertValidatedSqlNode(SqlToOperationConverter.java:345) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:238) ~[?:?]
    at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:105) ~[?:?]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$parseStatement$1(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:88) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.parseStatement(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    ... 11 more
Caused by: org.apache.flink.table.api.ValidationException: Invalid expression for column "a".
    at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.readAndValidateFieldExpression(FlinkFakerTableSourceFactory.java:187) ~[flink-faker-0.4.1.jar:?]
    at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.createDynamicTableSource(FlinkFakerTableSourceFactory.java:90) ~[flink-faker-0.4.1.jar:?]
    at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.createDynamicTableSource(FlinkFakerTableSourceFactory.java:21) ~[flink-faker-0.4.1.jar:?]
    at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:156) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:184) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.createDynamicTableSource(CatalogSourceTable.java:178) ~[?:?]
    at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:116) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3585) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2507) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2144) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2093) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3438) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570) ~[?:?]
    at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:198) ~[?:?]
    at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:190) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:1240) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:1188) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertValidatedSqlNode(SqlToOperationConverter.java:345) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:238) ~[?:?]
    at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:105) ~[?:?]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$parseStatement$1(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:88) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.parseStatement(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    ... 11 more
Caused by: java.lang.RuntimeException: Unable to resolve #{city_suffix} directive.
    at net.datafaker.service.FakeValuesService.resolveExpression(FakeValuesService.java:446) ~[flink-faker-0.4.1.jar:?]
    at net.datafaker.service.FakeValuesService.lambda$resolveExpression$1(FakeValuesService.java:450) ~[flink-faker-0.4.1.jar:?]
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_252]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_252]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_252]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_252]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_252]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_252]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_252]
    at net.datafaker.service.FakeValuesService.resolveExpression(FakeValuesService.java:457) ~[flink-faker-0.4.1.jar:?]
    at net.datafaker.service.FakeValuesService.expression(FakeValuesService.java:407) ~[flink-faker-0.4.1.jar:?]
    at net.datafaker.Faker.expression(Faker.java:713) ~[flink-faker-0.4.1.jar:?]
    at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.readAndValidateFieldExpression(FlinkFakerTableSourceFactory.java:185) ~[flink-faker-0.4.1.jar:?]
    at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.createDynamicTableSource(FlinkFakerTableSourceFactory.java:90) ~[flink-faker-0.4.1.jar:?]
    at com.github.knaufk.flink.faker.FlinkFakerTableSourceFactory.createDynamicTableSource(FlinkFakerTableSourceFactory.java:21) ~[flink-faker-0.4.1.jar:?]
    at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:156) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.factories.FactoryUtil.createDynamicTableSource(FactoryUtil.java:184) ~[flink-table-api-java-uber-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.createDynamicTableSource(CatalogSourceTable.java:178) ~[?:?]
    at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:116) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3585) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2507) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2144) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2093) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3438) ~[?:?]
    at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570) ~[?:?]
    at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:198) ~[?:?]
    at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:190) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:1240) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:1188) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertValidatedSqlNode(SqlToOperationConverter.java:345) ~[?:?]
    at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:238) ~[?:?]
    at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:105) ~[?:?]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$parseStatement$1(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.client.gateway.context.ExecutionContext.wrapClassLoader(ExecutionContext.java:88) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    at org.apache.flink.table.client.gateway.local.LocalExecutor.parseStatement(LocalExecutor.java:172) ~[flink-sql-client-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
    ... 11 more
knaufk commented 2 years ago

@bodiam could you have a look at this when you have time? Seems to be rather releated to data-faker than flink-faker, no?

bodiam commented 2 years ago

@knaufk don't know, not sure what you're doing here. Can you reproduce this without flink? Then we can more easily fix it

bodiam commented 2 years ago

@knaufk which version of datafaker is this? Also, could you try 1.3.0-SNAPSHOT if you're running an earlier version?

snuyanzin commented 2 years ago

@tsreaper as a WA currently it should work for Address.city (with first capital letter) like that 

create temporary table faker ( a string ) with ( 'connector' = 'faker', 'number-of-rows' = '10', 'fields.a.expression' = '#{Address.city}' );
select * from faker;
bodiam commented 2 years ago

@tsreaper Is this still an issue or can this be closed?