open-telemetry / opentelemetry-python

OpenTelemetry Python API and SDK
https://opentelemetry.io
Apache License 2.0
1.76k stars 614 forks source link

Configurable number of export attempts #3815

Open Blissbill opened 6 months ago

Blissbill commented 6 months ago

Hello Perhaps this question has already been asked, but after looking through all the issues I have not found anything similar. At the moment we are using grpc-exporter to collect code execution time. But sometimes it happens that the metrics collector is unavailable. Because of this, the exporter may stop the whole service until sending attempts are finished. The logs at this time show the following: Transient error StatusCode.UNAVAILABLE encountered while exporting traces, retrying in 32s. After looking at the source code, I noticed that we can't influence the number of attempts and delay time from the outside. In our case, one or two attempts are enough and if we fail to export, we are ready to move on rather than be distracted by meaningless retries with a long delay. The easiest option seems to be to do a max_value setting via the exporter constructor or from an environment variable. Or maybe I'm missing something and the number of attempts can be adjusted? Thanks

nykolas-bata commented 4 months ago

👋

We’re experiencing the same issue, although on the HTTP exporter. As a workaround we subclass and override the _MAX_RETRY_TIMEOUT, but it’d be preferable to have the retry behaviour be configurable.