Ability to handle publication failure

sebarys commented 4 years ago

Short description

As discussed on https://discuss.lightbend.com/t :

It could be beneficial to have version of Producer flow that will allow interpret kafka client error instead of failing stream in case of error (support emitting success/error downstream).

Details

We can introduce:

  sealed trait ResultsWithError[K, V, PassThrough] {
    def passThrough: PassThrough
  }

  ProducerMessage.PulbicationFailedResult(ex: Throwable, failedElement: Envelope[K, V, PassThrough]) extends ResultsWithError[K, V, PassThrough]

and extent subtypes of sealed trait Results[K, V, PassThrough] to also extend ResultsWithError

Such solution could enable scenarios like:

integration with legacy systems to ensure guaranteed publication - even in case of failure we can notify caller e.g. using attached as PassThrough element CompletableFuture
resilient KafkaProducer that don't discard messages that were “in the stream” when it restarts

ennru commented 4 years ago

Thank you for suggesting this. Please feel free to try it out and suggest the necessary changes in a pull request.

baltiyskiy commented 4 years ago

Possibly duplicate: https://github.com/akka/alpakka-kafka/issues/250

ennru commented 4 years ago

Similar request in #253

akka / alpakka-kafka

Ability to handle publication failure #1101

Short description

Details