search

akka / alpakka-kafka

Alpakka Kafka connector - Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.
https://doc.akka.io/libraries/alpakka-kafka/current/
Other
1.42k stars 386 forks source link

[Failure] CommittingSpec: Multiple consumers to one committer... #957

Open ennru opened 5 years ago

ennru commented 5 years ago

...must merge multiple consumers` offsets for different partitions (#942) 

[info] Multiple consumers to one committer
[info] - must merge multiple consumers` offsets for different partitions (#942) *** FAILED *** (19 seconds, 660 milliseconds)
[info]   The future returned an exception of type: akka.kafka.CommitTimeoutException, with message: Kafka commit took longer than: 15 seconds. (CommittingSpec.scala:459)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.concurrent.Futures$FutureConcept.tryTryAgain$1(Futures.scala:531)
[info]   at org.scalatest.concurrent.Futures$FutureConcept.futureValueImpl(Futures.scala:550)
[info]   at org.scalatest.concurrent.Futures$FutureConcept.futureValueImpl$(Futures.scala:479)
[info]   at org.scalatest.concurrent.ScalaFutures$$anon$1.futureValueImpl(ScalaFutures.scala:275)
[info]   at org.scalatest.concurrent.Futures$FutureConcept.futureValue(Futures.scala:476)
[info]   at org.scalatest.concurrent.Futures$FutureConcept.futureValue$(Futures.scala:475)
[info]   at org.scalatest.concurrent.ScalaFutures$$anon$1.futureValue(ScalaFutures.scala:275)
[info]   at akka.kafka.scaladsl.CommittingSpec.$anonfun$new$44(CommittingSpec.scala:459)
[info]   at akka.stream.testkit.scaladsl.StreamTestKit$.assertAllStagesStopped(StreamTestKit.scala:32)
nikkatalnikov commented 5 years ago

we got this error with committablePartitionedSource. any progress on it?

seglo commented 4 years ago

https://travis-ci.org/akka/alpakka-kafka/jobs/619520795#L556

ennru commented 4 years ago

https://travis-ci.org/akka/alpakka-kafka/jobs/625389283#L983 in #1005 https://travis-ci.org/akka/alpakka-kafka/jobs/625389285#L986 in #1005

ennru commented 4 years ago

https://travis-ci.org/akka/alpakka-kafka/jobs/625607991#L978 in #926

ennru commented 4 years ago

I'm trying to understand how this can be a 15s timeout https://travis-ci.org/akka/alpakka-kafka/jobs/625654499#L979-L982 So it must be that the actor is already terminated when the commit is sent.

ennru commented 4 years ago

It is indeed that the first consumer actor is terminated before the commit message is sent to it. This is the scenario where it is important to have a stop-timeout which is large enough to keep the actor around after downstream completed.

seglo commented 4 years ago

Hopefully, this is resolved in #1007, but I'll keep the issue open for the time being.