prestodb / presto

The official home of the Presto distributed SQL query engine for big data
http://prestodb.io
Apache License 2.0
15.85k stars 5.32k forks source link

presto-kafka CSV Decoder Can't find Mapping for column handle KafkaColumnHandle #6694

Closed getGitxxx closed 5 years ago

getGitxxx commented 7 years ago

hi,everybody:

I'm trying to query SQL on presto-kafka with CSV format topic, table config called "db.tab1.json" goes here:

{
  "tableName": "tab1",
  "schemaName": "db",
  "topicName": "tag_topic",
  "key": {
    "dataFormat": "raw",
    "fields": [
      {
        "name": "kafka_key",
        "dataFormat": "LONG",
        "type": "BIGINT",
        "hidden": "false"
      }
    ]
  },
  "message": {
    "dataFormat": "csv",
    "fields": [
      {
        "name":"time_stamp",
        "mapping":0,
        "type":"VARCHAR"
      },
      {
        "name":"uid",
        "mapping":1,
        "type":"VARCHAR"
      }
...
]

I saved config file and restarted the Presto cluster, there are new catalog,schema,tables about kafka's Topic can been seen. But when I execute the query like "select * from db.tab1 limit 10",I got error says:"No mapping for column handle KafkaColumnHandle " Detail info:

java.lang.IllegalStateException: No mapping for column handle KafkaColumnHandle{connectorId=kafka, ordinalPosition=0, name=kafka_key, type=bigint, mapping=null, dataFormat=LONG, formatHint=null, keyDecoder=true, hidden=false, internal=false}!
    at com.google.common.base.Preconditions.checkState(Preconditions.java:197)
    at com.facebook.presto.decoder.csv.CsvRowDecoder.decodeRow(CsvRowDecoder.java:76)
    at com.facebook.presto.kafka.KafkaRecordSet$KafkaRecordCursor.nextRow(KafkaRecordSet.java:222)
    at com.facebook.presto.kafka.KafkaRecordSet$KafkaRecordCursor.advanceNextPosition(KafkaRecordSet.java:175)
    at com.facebook.presto.spi.RecordPageSource.getNextPage(RecordPageSource.java:99)
    at com.facebook.presto.operator.TableScanOperator.getOutput(TableScanOperator.java:246)
    at com.facebook.presto.operator.Driver.processInternal(Driver.java:378)
    at com.facebook.presto.operator.Driver.processFor(Driver.java:301)
    at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:622)
    at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:529)
    at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:665)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

How to let this feature work correctly? Can anybody would like to give me a little suggestion about this issue? Thanks for any reply!

cawallin commented 7 years ago

@getGitxxx Sorry for the late response. It seems like the kafka connector isn't installed properly, do you have kafka.properties and the table config on all of the Presto nodes?

lifehacking commented 7 years ago

me too

failed: No mapping for column handle KafkaColumnHandle{connectorId=kafka, ordinalPosition=0, name=kafka_key, type=bigint, mapping=null, dataFormat=LONG, formatHint=null, keyDecoder=true, hidden=false, internal=false}!

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had any activity in the last 2 years. If you feel that this issue is important, just comment and the stale tag will be removed; otherwise it will be closed in 7 days. This is an attempt to ensure that our open issues remain valuable and relevant so that we can keep track of what needs to be done and prioritize the right things.