apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.05k stars 1.82k forks source link

[Bug] [Module Name] hbase sink to starrocks, hbase column family can not transform correctly on filedmapper. #7707

Closed ilovd closed 2 weeks ago

ilovd commented 2 months ago

Search before asking

What happened

Hbase Sink to Starrocks,the configuration of transform.fieldmapper info:name doesn't work.

SeaTunnel Version

2.3.6

SeaTunnel Config

env {
  parallelism = 1
}

source {
  Hbase {
    zookeeper_quorum = "10.53.14.206:2181" 
    table = "users" 
    caching = 1000 
    batch = 100 
    cache_blocks = false 
    query_columns = ["rowkey", "info:name", "info:age"]
    result_table_name = "fake"
    schema = {
      columns = [
        { 
          name = "rowkey" 
          type = string 
        },
        {
          name = "info:name"
          type = string
        },
        {
          name = "info:age" 
          type = int
        }
      ]
    }
  }
}

transform {
  FieldMapper {
    source_table_name = "fake"
    result_table_name = "fake1"
    field_mapper = {
        rowkey = code
        info:name = data
        info:age = msg
    }
  }
}

sink {
  Starrocks {
        source_table_name = "fake1"
    nodeUrls = ["10.53.14.206:8030"]
        username = admin
        password = "111111"
        database = "test"
        table = "hbasetest"
        starrocks.config {
            format = "json"
        }
}

Running Command

bin/seatunnel.sh -c config/hbase.conf -m local

Error Exception

There is no filed info:name from the upstream source.

Zeta or Flink or Spark Version

zeta

Java or Scala Version

jdk11

Screenshots

transform can not recognize the field "info:name", "info:age"。 i want to mapping hbase column to starrocks column, but failed. If i only use field rowkey to mapping, it will success. but when i mapping more columns , it will failed.

Are you willing to submit PR?

Code of Conduct

ilovd commented 1 month ago

image

ilovd commented 1 month ago

when i use sql transform: image

error msg: image

github-actions[bot] commented 3 weeks ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 2 weeks ago

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.