nur858 commented 5 years ago

Description

I am running .net core kafka consumer in container. I have 10 workers in consumer group. It works well for the most part except occasionally all consumers go silent with the following log.

June 19th 2019, 11:42:55.552 | OnError :: ErrorCode:Local_TimedOut, IsBrokerError:False, IsLocalError:True, Reason:GroupCoordinator: 1 request(s) timed out: disconnect (after 6545216ms in state UP)

  | June 19th 2019, 11:42:44.455 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-7gxm9#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out JoinGroupRequest in flight (after 183921ms, timeout #0) Facility: REQTMOUT  Level: Notice
  | June 19th 2019, 11:42:44.455 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-7gxm9#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests Facility: REQTMOUT  Level: Warning

How to reproduce

Checklist

Please provide the following information:

[ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
[x] Confluent.Kafka nuget version. 1.0.1.1
[ ] Apache Kafka version.
[x] Client configuration. EnableAutoCommit = true, EnableAutoOffsetStore = false, AutoOffsetReset = 'Latest', MaxPollIntervalMs = 3 601000, SessionTimeoutMs = 60 1000, HeartbeatIntervalMs = 20 1000, QueuedMaxMessagesKbytes = 10240, // 10 MB SocketKeepaliveEnable = true, Acks = Acks.Leader, LogConnectionClose = false, MaxInFlight = 1, Debug = "consumer,cgrp"
[x] Operating system. CentOS

[ ] Provide logs (with "debug" : "..." as necessary in configuration).


June 19th 2019, 11:42:55.552 | OnError :: ErrorCode:Local_TimedOut, IsBrokerError:False, IsLocalError:True, Reason:GroupCoordinator: 1 request(s) timed out: disconnect (after 6545216ms in state UP)

| June 19th 2019, 11:42:44.455 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-7gxm9#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out JoinGroupRequest in flight (after 183921ms, timeout #0) Facility: REQTMOUT Level: Notice

| June 19th 2019, 11:42:44.455 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-7gxm9#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests Facility: REQTMOUT Level: Warning

| June 19th 2019, 11:42:44.454 | OnError :: ErrorCode:Local_TimedOut, IsBrokerError:False, IsLocalError:True, Reason:GroupCoordinator: 1 request(s) timed out: disconnect (after 6526990ms in state UP)

| June 19th 2019, 11:42:44.379 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-cbzqt#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out JoinGroupRequest in flight (after 183882ms, timeout #0) Facility: REQTMOUT Level: Notice

| June 19th 2019, 11:42:44.379 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-cbzqt#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests Facility: REQTMOUT Level: Warning

| June 19th 2019, 11:42:44.379 | OnError :: ErrorCode:Local_TimedOut, IsBrokerError:False, IsLocalError:True, Reason:GroupCoordinator: 1 request(s) timed out: disconnect (after 6557039ms in state UP)

| June 19th 2019, 11:42:44.307 | OnError :: ErrorCode:Local_TimedOut, IsBrokerError:False, IsLocalError:True, Reason:GroupCoordinator: 1 request(s) timed out: disconnect (after 6527676ms in state UP)

| June 19th 2019, 11:42:44.307 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-w9h6l#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out JoinGroupRequest in flight (after 183953ms, timeout #0) Facility: REQTMOUT Level: Notice

| June 19th 2019, 11:42:44.306 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-w9h6l#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests Facility: REQTMOUT Level: Warning

| June 19th 2019, 11:42:44.255 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-rnscz#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out JoinGroupRequest in flight (after 183656ms, timeout #0) Facility: REQTMOUT Level: Notice

| June 19th 2019, 11:42:44.255 | OnError :: ErrorCode:Local_TimedOut, IsBrokerError:False, IsLocalError:True, Reason:GroupCoordinator: 1 request(s) timed out: disconnect (after 6552628ms in state UP)

| June 19th 2019, 11:42:44.254 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-rnscz#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests Facility: REQTMOUT Level: Warning

| June 19th 2019, 11:42:44.216 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-w7v7p#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests Facility: REQTMOUT Level: Warning

| June 19th 2019, 11:42:44.213 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-w7v7p#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out JoinGroupRequest in flight (after 183695ms, timeout #0) Facility: REQTMOUT Level: Notice

| June 19th 2019, 11:42:44.196 | OnError :: ErrorCode:Local_TimedOut, IsBrokerError:False, IsLocalError:True, Reason:GroupCoordinator: 1 request(s) timed out: disconnect (after 6529858ms in state UP)

| June 19th 2019, 11:42:44.195 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-vd5px#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out JoinGroupRequest in flight (after 183623ms, timeout #0) Facility: REQTMOUT Level: Notice

| June 19th 2019, 11:42:44.195 | OnLog :: Name: delivery-guarantee-worker-59b98c8c7-vd5px#consumer-1 Message: [thrd:GroupCoordinator]: GroupCoordinator/3: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests Facility: REQTMOUT Level: Warning


 
 - [ ] Provide broker log excerpts.
 - [x] Critical issue.

nur858 commented 5 years ago

Including more log data in case if that is helpful. No message is being processed. What could possibly cause frequent unassign events?

OnLog :: Name: delivery-guarantee-worker-6674cc9f87-nhqdt#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:20.931    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-qjd7f#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:20.358    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-qjd7f#consumer-1 Message: [thrd:main]: Group "CatalogService" received op GET_ASSIGNMENT (v0) in state up (join state wait-join, v10 vs 0) Facility: CGRPOP  Level: Debug
    June 21st 2019, 16:07:16.670    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-tdrcr#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:15.001    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-555vv#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:13.732    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-xpjr4#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:13.731    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-xpjr4#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:13.731    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-xpjr4#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:12.626    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-ht4ff#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:12.626    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-ht4ff#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:12.626    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-ht4ff#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:11.823    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-2hq7k#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:11.238    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gzfn5#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:11.178    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gzfn5#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:11.027    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-nhqdt#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:11.027    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-nhqdt#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:11.027    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-nhqdt#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:10.930    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-qjd7f#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:10.930    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-qjd7f#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:10.930    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-qjd7f#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:10.853    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-2hq7k#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:10.853    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-2hq7k#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:09.217    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gbr9v#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:09.168    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gbr9v#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:09.146    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-vdhqk#consumer-1 Message: [thrd:main]: kafkac3n7.host.bo1.csnzoo.com:9092/7: Group "CatalogService" coordinator is kafkac3n3.host.bo1.csnzoo.com:9092 id 3 Facility: CGRPCOORD  Level: Debug
    June 21st 2019, 16:07:08.974    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-vdhqk#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:08.640    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-vdhqk#consumer-1 Message: [thrd:main]: Group "CatalogService" received op GET_ASSIGNMENT (v0) in state up (join state wait-join, v10 vs 0) Facility: CGRPOP  Level: Debug
    June 21st 2019, 16:07:08.593    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-tdrcr#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:08.306    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-tdrcr#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:08.091    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-tdrcr#consumer-1 Message: [thrd:main]: Group "CatalogService" received op GET_ASSIGNMENT (v0) in state up (join state wait-join, v10 vs 0) Facility: CGRPOP  Level: Debug
    June 21st 2019, 16:07:06.904    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-555vv#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:06.526    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-555vv#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:05.772    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-555vv#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:04.629    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-555vv#consumer-1 Message: [thrd:main]: Group "CatalogService" received op GET_ASSIGNMENT (v0) in state up (join state wait-join, v11 vs 0) Facility: CGRPOP  Level: Debug
    June 21st 2019, 16:07:03.735    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-xpjr4#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:03.731    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-xpjr4#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:03.729    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-xpjr4#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:02.627    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-ht4ff#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:02.626    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-ht4ff#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:02.625    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-ht4ff#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:01.177    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gzfn5#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:01.177    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gzfn5#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:01.176    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gzfn5#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:01.027    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-nhqdt#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:01.027    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-nhqdt#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:01.026    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-nhqdt#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:00.974    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-qjd7f#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:00.930    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-qjd7f#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:00.930    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-qjd7f#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:00.926    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-2hq7k#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:07:00.854    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-2hq7k#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:07:00.853    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-2hq7k#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:58.749    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gbr9v#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:58.749    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gbr9v#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:06:58.749    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gbr9v#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:57.756    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-vdhqk#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:57.755    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-vdhqk#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:06:56.670    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-tdrcr#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:56.669    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-tdrcr#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:55.000    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-555vv#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:06:54.999    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-555vv#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:53.731    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-xpjr4#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:53.730    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-xpjr4#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:53.730    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-xpjr4#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:06:53.623    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-ht4ff#consumer-1 Message: [thrd:main]: kafkac3n12.host.bo1.csnzoo.com:9092/12: Group "CatalogService" coordinator is kafkac3n3.host.bo1.csnzoo.com:9092 id 3 Facility: CGRPCOORD  Level: Debug
    June 21st 2019, 16:06:53.622    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-ht4ff#consumer-1 Message: [thrd:main]: kafkac3n12.host.bo1.csnzoo.com:9092/12: Group "CatalogService": querying for coordinator: intervaled in state up Facility: CGRPQUERY  Level: Debug
    June 21st 2019, 16:06:52.626    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-ht4ff#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:06:52.625    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-ht4ff#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:52.296    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-nhqdt#consumer-1 Message: [thrd:main]: kafkac3n5.host.bo1.csnzoo.com:9092/5: Group "CatalogService": querying for coordinator: intervaled in state up Facility: CGRPQUERY  Level: Debug
    June 21st 2019, 16:06:52.194    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-ht4ff#consumer-1 Message: [thrd:main]: Group "CatalogService" received op GET_ASSIGNMENT (v0) in state up (join state wait-join, v11 vs 0) Facility: CGRPOP  Level: Debug
    June 21st 2019, 16:06:52.171    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gzfn5#consumer-1 Message: [thrd:main]: kafkac3n12.host.bo1.csnzoo.com:9092/12: Group "CatalogService" coordinator is kafkac3n3.host.bo1.csnzoo.com:9092 id 3 Facility: CGRPCOORD  Level: Debug
    June 21st 2019, 16:06:51.851    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-2hq7k#consumer-1 Message: [thrd:main]: kafkac3n12.host.bo1.csnzoo.com:9092/12: Group "CatalogService": querying for coordinator: intervaled in state up Facility: CGRPQUERY  Level: Debug
    June 21st 2019, 16:06:51.176    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gzfn5#consumer-1 Message: [thrd:main]: OffsetCommit internal error: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:51.176    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gzfn5#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:51.027    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-nhqdt#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:50.930    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-qjd7f#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:06:50.929    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-qjd7f#consumer-1 Message: [thrd:main]: OffsetCommit for -1 partition(s): cgrp auto commit timer: returned: Local: No offset stored Facility: COMMIT  Level: Debug
    June 21st 2019, 16:06:50.854    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-2hq7k#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:06:50.787    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gzfn5#consumer-1 Message: [thrd:main]: Group "CatalogService" received op GET_ASSIGNMENT (v0) in state up (join state wait-join, v11 vs 0) Facility: CGRPOP  Level: Debug
    June 21st 2019, 16:06:50.401    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-nhqdt#consumer-1 Message: [thrd:main]: Group "CatalogService" received op GET_ASSIGNMENT (v0) in state up (join state wait-join, v10 vs 0) Facility: CGRPOP  Level: Debug
    June 21st 2019, 16:06:50.356    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-qjd7f#consumer-1 Message: [thrd:main]: Group "CatalogService" received op GET_ASSIGNMENT (v0) in state up (join state wait-join, v10 vs 0) Facility: CGRPOP  Level: Debug
    June 21st 2019, 16:06:48.752    OnLog :: Name: delivery-guarantee-worker-6674cc9f87-gbr9v#consumer-1 Message: [thrd:main]: Group "CatalogService": unassign done in state up (join state wait-join): without new assignment: OffsetCommit done (__NO_OFFSET) Facility: UNASSIGN  Level: Debug
    June 21st 2019, 16:06:48.508

erik-neumann commented 5 years ago

I think we're having the same issue. Looks like the consumer's group state get somehow caught in a loop with an invalid state and does not recover until to restart the client (or trigger a reconnect). I opened an issue at librdkafka (edenhill/librdkafka#2363) but did not get any feedback yet.

nur858 commented 5 years ago

I was able to resolve the issue by using unique consumer group name. I have 3 consumer clusters consuming from 3 different topics who are created in the same broker cluster.All three clusters were sharing the same consumer group name. As soon as I gave unique consumer group name for each cluster, everything started working. I am not sure if this is by design.

erik-neumann commented 5 years ago

Thanks for the hint. But actually we have a 3 broker cluster and the consumers do consume from different topics and do not share the same groupId already. Your case is interesting. This means that using the same groupId but different topic somehow runs into a conflict...

jeffjnh commented 3 years ago

Any insight into a fix for this?

anchitj commented 2 weeks ago

Is this still an issue?

confluentinc / confluent-kafka-dotnet

Consumer temporarily stops message consumption #985

Description

How to reproduce

Checklist