Error events can silently be lost

[svix-aaron1011]

Environment

sentry v0.31.5

Steps to Reproduce

There are multiple ways that Sentry can silently lose error events:

• When any of the sentry transport backends fail to send a message, they only log this at debug level: https://github.com/getsentry/sentry-rust/blob/10c56404e25c41b2e194ed27126fb5a5b9cd0697/sentry/src/transports/reqwest.rs#L99 If the off-by-default feature debug-logs is not enabled, then this won't get logged at all unless Sentry is configured with ClientOptions.debug = true (also off-by-default).
• The TransportThread used for sending messages uses a channel with a size of 30. If this size is exceeded, then events will be dropped (with the same debug-logging issue as above): https://github.com/getsentry/sentry-rust/blob/10c56404e25c41b2e194ed27126fb5a5b9cd0697/sentry/src/transports/tokio_thread.rs#L88-L93

Expected Result

When using Sentry in production, it can be critical that any errors are either reported to sentry, or trigger some kind of error/panic indicating that an event was not reported.

It would be very useful for Sentry to:

• Log these kinds of messages as errors, rather than off-by-default debug messages
• Provide the ability to use an unbounded channel (or use an unbounded channel for error/exception events), to prevent events from getting dropped in the first place

[loewenheim]

Thank you for the report, we'll discuss it.

[Swatinem]

There is also the possibility of this being dropped in the transport due to rate limits and some such.

Ideally, all the capturing methods would a Future that resolves to a descriptive enum with various variants for the different reasons a message/exception might be lost. However, that would be quite deep change to how things are working right now, and would require lots of effort to make happen.

I don’t think the idea of an unbounded channel, or different channels for different event types makes sense. The transports are completely agnostic to that, as they are only dealing with opaque envelopes.