behavior.on.malformed.documents set to report causes task failure

[franfreire]

I am attempting to configure a Dead Letter Queue (DLQ) for handling malformed documents. My expectation was that setting behavior.on.malformed.documents to report would allow the task to continue processing new records even if some documents are malformed and sent to the DLQ. However, despite the records appearing in my DLQ as anticipated, the task transitions to a failed state and halts, preventing the submission of additional records to OpenSearch.

Is this behavior by design? If so, is there a workaround or setting that enables the continuation of data transmission to OpenSearch while still routing malformed documents to the DLQ?

Thank you for your assistance.

[reta]

Hi @franfreire

Is this behavior by design?

No, it is not, your understanding is correct: the malformed document should be reported and not indexed, but task should not fail.

If so, is there a workaround or setting that enables the continuation of data transmission to OpenSearch while still routing malformed documents to the DLQ?

I suspect you may have another issue here, beside just some documents, fe the whole bulk update may fail - that would cause the task to fail as well, could you please share if there are any exceptions in logs?

Thank you.

[franfreire]

Hello,

After further investigation and inspecting the error message shown by the task in the /status endpoint, I discovered that the error was "Tolerance exceeded in error handler". I managed to resolve this issue by setting "errors.tolerance": "all" in my connector configuration.

I am sharing this solution in the hope that it might assist anyone facing a similar situation. You can close the issue.

Thank you for your time and assistance.