omhttp silently discards data after http 403 response

orion84 commented 3 years ago

When events are forwarded using omhttp and the destination responds with http status 403 any following events entering the omhttp action are silently discarded and endpoint is never contacted again. Causing data loss and even after fixing the issue at the endpoint that was causing the 403, rsyslog needs a restart to start sending again.

See more detailed description below.

Expected behavior

An event comes in to rsyslog, is passed to omhttp for delivery. omhttp connects to Splunk HTTP Event Collector endpoint to deliver the data and receives a 200 response.

splunk admin disables the respective token on HEC endpoint.

An event comes in to rsyslog, is passed to omhttp for delivery. omhttp connects to Splunk HTTP Event Collector endpoint to deliver the data and receives a 403 failure response (Token Disabled).

splunk admin enables token again.

Next message comes in (or previous message is retried, depending on retry settings). omhttp connects to HTTP endpoint to deliver the data and receives a 200 success response

Actual behavior

An event comes in to rsyslog, is passed to omhttp for delivery. omhttp connects to Splunk HTTP Event Collector endpoint to deliver the data and receives a 200 response.

splunk admin disables the respective token on HEC endpoint.

An event comes in to rsyslog, is passed to omhttp for delivery. omhttp connects to Splunk HTTP Event Collector endpoint to deliver the data and receives a 403 failure response (Token Disabled).

splunk admin enables token again.

Next message comes in (or previous message is retried, depending on retry settings). omhttp does not even try to connect to endpoint, instead just fails (silently?): ../action.c: doTransaction: action 0, processing msg 0, result -2218

Steps to reproduce the behavior

Configure and HTTP server to accept messages from rsyslog Setup rsyslog with some event source and an action using omhttp. Configure HTTP server to respond with 403 status Configure HTTP server to accept messages again

Environment

rsyslog version: 8.2102.0
platform: centos7

omhttp action config used:

action(
                name="tcp9001-omhttptest-HEC"
                type="omhttp"
                server="***"
                serverport="8088"
                useHttps = "on"
                tls.cacert = "/opt/rsyslog/ca.pem"
                dynrestpath = "off"
                restpath="services/collector/event?auto_extract_timestamp=true"
                httpheaders=[
                        "Authorization: Splunk ****"
                ]
                template="eventhec_omhttptest"

                batch="off"
                action.resumeRetryCount="-1"
              )

See debug log with few successful messages then 403 response then silent failures attached: omhttp 403 response issue.log

davidelang commented 3 years ago

does a HUP to rsyslog get it to retry? that tells it to close and reopen all outputs and I would expect it to attempt to deliver again.

I don't disagree that there should be an option to make delivery errors block delivery rather than drop the message, but I would expect a HUP to resume delivery.

David Lang

On Wed, 14 Jul 2021, orion84 wrote:

Date: Wed, 14 Jul 2021 07:17:20 -0700 From: orion84 @.> Reply-To: rsyslog/rsyslog @.> To: rsyslog/rsyslog @.> Cc: Subscribed @.> Subject: [rsyslog/rsyslog] omhttp silently discards data after http 403 response (#4636)

When events are forwarded using omhttp and the destination responds with http status 403 any following events entering that action are silently discarded and endpoint is never contacted again. Causing data loss and even after fixing the issue at the endpoint that was causing the 403, rsyslog needs a restart to start sending again.

See more detailed description below.

Expected behavior

An event comes in to rsyslog, is passed to omhttp for delivery. omhttp connects to Splunk HTTP Event Collector endpoint to deliver the data and receives a 200 response.

splunk admin disables the respective token on HEC endpoint.

An event comes in to rsyslog, is passed to omhttp for delivery. omhttp connects to Splunk HTTP Event Collector endpoint to deliver the data and receives a 403 failure response (Token Disabled).

splunk admin enables token again.

Next message comes in (or previous message is retried, depending on retry settings). omhttp connects to HTTP endpoint to deliver the data and receives a 200 success response

Actual behavior

An event comes in to rsyslog, is passed to omhttp for delivery. omhttp connects to Splunk HTTP Event Collector endpoint to deliver the data and receives a 200 response.

splunk admin disables the respective token on HEC endpoint.

An event comes in to rsyslog, is passed to omhttp for delivery. omhttp connects to Splunk HTTP Event Collector endpoint to deliver the data and receives a 403 failure response (Token Disabled).

splunk admin enables token again.

Next message comes in (or previous message is retried, depending on retry settings). omhttp does not even try to connect to endpoint, instead just fails (silently?): ../action.c: doTransaction: action 0, processing msg 0, result -2218

Steps to reproduce the behavior

Configure and HTTP server to accept messages from rsyslog Setup rsyslog with some event source and an action using omhttp. Configure HTTP server to respond with 403 status Configure HTTP server to accept messages again

Environment

rsyslog version: 8.2102.0

platform: centos7

omhttp action config used: action( name="tcp9001-omhttptest-HEC" type="omhttp" server="*" serverport="8088" useHttps = "on" tls.cacert = "/opt/rsyslog/ca.pem" dynrestpath = "off" restpath="services/collector/event?auto_extract_timestamp=true" httpheaders=[ "Authorization: Splunk **" ] template="eventhec_omhttptest"
           batch="off"
           action.resumeRetryCount="-1"
         )
See debug log with few successful messages then 403 response then silent failures attached: omhttp 403 response issue.log

orion84 commented 3 years ago

Just tested with HUP, that doesn't pull it from its datafail (error 2218) state.

rgerhards commented 3 years ago

Thx for the info. Please note that this is a contributed module, so it may help to try to reach the author directly. Nevertheless, I'll try to have a look when I have time. But that'll be August for sure.

orion84 commented 11 months ago

@rgerhards I take it you have not yet had time to look into this? We ran into similar issues again now (with 404 this time) which reminded me of this issue ticket.

@ctramnitz any thoughts on this?