Fails silently when streaming batches of un-enriched events

[amrhassan]

Trying to stream objects that don't correspond to the structure of EventEnriched a fails silently and returns batches with Nothing in the _events field.

[mtesseract]

For the record: the streaming APIs exposed by nakadi-client indeed (currently?) drop events that coudl not be deserialized. The reason behind this is that from a resilience perspective perspective a single event that failed to conform to some schema, should not bring down the whole service. Instead, what is additionally exposed is a mechanism to set a callback that is called on deserialization failures (setDeserializationFailureCallback). Using this, the user can handle the failure in whatever way that is desired.

It is debatable whether this is the right thing to do — maybe we should keep this issue open and exchange pros and cons?

Possible alternatives include:

1. Fail and terminate stream consumption on deserialization failures
2. Fail and terminate stream consumption EXCEPT if a deserialization failure callback is registered

Actually, I like the second idea. This would mean: Less surprises caused by silently ignoring errors. But IF the user knows about these callbacks, they can be installed and imply a different failure mechanism. Addtionally, using a return value, it could be decided if the failed events should simply be dropped or if some exception should be propagated upwards.

What do you think, @amrhassan ?

[amrhassan]

I see your point about resilience, and I agree more about your second alternative of making it opt-in. I would always expect the default behaviour to not silently swallow up errors.

[mtesseract]

This is done.