Application crashes while rebalancing

[ogirard]

Description

We have a topology with different joins (KStream,KTable and KStream,KStream) and map/filter/rebalance steps in-between. While running the streaming app in k8s, we were not able to scale it up to 10 consumers. We tried to scale it right after deploy while it was reading records and initializing state (in-memory stores only) from fairly large topics (>5M records of ~500b). Right after the scaling, the application transitioned to ERROR state and was shut down for the main part of the consumers.

Exception

[10:05:13 ERR Streamiz.Kafka.Net.Processors.StreamThread] stream-thread[team-first-0c72d76c-e583-406e-ac65-8e06fd2e9ae3-stream-thread-0] Encountered the following error during processing:
System.NullReferenceException: Task was unexpectedly missing for partition poc-inputA-repartition [[0]]
   at Streamiz.Kafka.Net.Processors.StreamThread.AddToTasks(IEnumerable`1 records)
   at Streamiz.Kafka.Net.Processors.StreamThread.Run()
[10:05:13 INF Streamiz.Kafka.Net.Processors.StreamThread] stream-thread[poc-0c72d76c-e583-406e-ac65-8e06fd2e9ae3-stream-thread-0] Shutting down
[10:05:13 INF Streamiz.Kafka.Net.Processors.StreamThread] stream-thread[poc-0c72d76c-e583-406e-ac65-8e06fd2e9ae3-stream-thread-0] Shutdown complete
[10:05:13 INF Streamiz.Kafka.Net.Processors.StreamThread] stream-thread[poc-0c72d76c-e583-406e-ac65-8e06fd2e9ae3-stream-thread-0] State transition from PENDING_SHUTDOWN to DEAD
[10:05:13 INF Streamiz.Kafka.Net.KafkaStream] stream-application[poc] State transition from RUNNING to ERROR
[10:05:13 ERR Streamiz.Kafka.Net.Processors.StreamStateManager] All stream threads have died. The instance will be in error state and should be closed

Tested with Streamiz v1.4.1 on .NET 6 in a Linux Container (base image mcr.microsoft.com/dotnet/aspnet:6.0).

[LGouellec]

Hi @ogirard,

Thanks for your issue. If you set the MaxPollRecords to 1, the issue still happening or not ?

config.MaxPollRecords = 1;

Best regards,