search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.36k stars 3.22k forks source link

routine load kafka property 未生效 #6138

Closed wenbaoup closed 3 years ago

wenbaoup commented 3 years ago

Describe the bug A clear and concise description of what the bug is. routine load kafka property "property.isolation.level"="read_committed" 无效 To Reproduce Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior A clear and concise description of what you expected to happen. 希望只读kafak中commit的数据

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Smartphone (please complete the following information):

Additional context Add any other context about the problem here.

stalary commented 3 years ago

let me see see

morningman commented 3 years ago

The version of the librdkafka we currently used does not support this property...

stalary commented 3 years ago

@morningman https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#compatibility It looks like we are also forward compatible after we upgrade to 1.0 isolation.level is generally used to exactly once in stream processing. https://github.com/edenhill/librdkafka/releases/tag/v1.2.0 we need upgrade v1.2.0 support consumer transaction

WindyGao commented 3 years ago

@morningman @stalary it's related with issue-3682 and issue-5485 after some research, the rdKafka client cames to the following conclusions

that said, it's theoretically possible to upgrade doris's rdKafka client to v1.4 or higher I will be testing against v1.4~v1.6 of the rdKafka client for routine load job

WindyGao commented 3 years ago

v1.2

@morningman https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#compatibility It looks like we are also forward compatible after we upgrade to 1.0 isolation.level is generally used to exactly once in stream processing. https://github.com/edenhill/librdkafka/releases/tag/v1.2.0 we need upgrade v1.2.0 support consumer transaction

yeah v1.2 is enough actually