Redis sink - unexpected error type, broken pipe

hungpr0 commented 1 year ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hello,

We are migration from Logstash to Vector and found logs aren't sent to redis randomly on Vector. It seems, problem is on redis sink. Please check debug log below.

From the docs:

Retry policy
Vector will retry failed requests (status == 429, >= 500, and != 501). Other responses will not be retried. You can control the number of retry attempts and backoff rate with the request.retry_attempts and request.retry_backoff_secs options.

Is there a way to force vector retry on broken pipe error?

Configuration

sinks:
  mysql_slow_log_sink:
    data_type: list
    encoding:
      codec: json
    endpoint: redis://10.0.0.111/0
    inputs:
    - mysql_slow_log_parser
    key: logstash-slowmysql
    type: redis

Version

vector 0.33.0 (x86_64-unknown-linux-gnu 89605fb 2023-09-27 14:18:24.180809939)

Debug Output

Oct 25 15:10:59 my_server vector[21771]: 2023-10-25T15:10:59.233501Z ERROR sink{component_kind="sink" component_id=mysql_slow_log_sink component_type=redis component_name=mysql_slow_log_sink}:request{request_id=32}: vector::sinks::util::retries: Unexpected error type; dropping the request. error=Error sending query: broken pipe internal_log_rate_limit=true
Oct 25 15:10:59 my_server vector[21771]: 2023-10-25T15:10:59.233553Z  WARN sink{component_kind="sink" component_id=mysql_slow_log_sink component_type=redis component_name=mysql_slow_log_sink}:request{request_id=32}: vector::sinks::util::adaptive_concurrency::controller: Unhandled error response. error=Error sending query: broken pipe internal_log_rate_limit=true
Oct 25 15:10:59 my_server vector[21771]: 2023-10-25T15:10:59.233568Z ERROR sink{component_kind="sink" component_id=mysql_slow_log_sink component_type=redis component_name=mysql_slow_log_sink}:request{request_id=32}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(SendError { source: broken pipe }) request_id=32 error_type="request_failed" stage="sending" internal_log_rate_limit=true
Oct 25 15:10:59 my_server vector[21771]: 2023-10-25T15:10:59.233595Z ERROR sink{component_kind="sink" component_id=mysql_slow_log_sink component_type=redis component_name=mysql_slow_log_sink}:request{request_id=32}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true



### Example Data

_No response_

### Additional Context

_No response_

### References

_No response_

neuronull commented 1 year ago

Hello,

This is odd. Definitely seems like we should be retrying that. This bit

vector::sinks::util::retries: Unexpected error type; dropping the request.

, occurs when we weren't able to downcast_ref the error type into a recognizable one to determine if we even should retry/not retry, so we default to drop/no retry.

Similar issue occurred in https://github.com/vectordotdev/vector/issues/6282

It seems that from the redis crate error types, this boils down to an i/o error (broken pipe) so the connection closed on us while trying to transmit the request.

hungpr0 commented 1 year ago

Hello,

This is odd. Definitely seems like we should be retrying that. This bit

vector::sinks::util::retries: Unexpected error type; dropping the request.

, occurs when we weren't able to downcast_ref the error type into a recognizable one to determine if we even should retry/not retry, so we default to drop/no retry.

Similar issue occurred in #6282

It seems that from the redis crate error types, this boils down to an i/o error (broken pipe) so the connection closed on us while trying to transmit the request.

So this can be fixed on vector side? Thanks.

hungpr0 commented 1 year ago

Just want to update. Problem is still on vector 0.33.1 (x86_64-unknown-linux-gnu 3cc27b9 2023-10-30 16:50:49.747931844) version

vectordotdev / vector