Open wguanicedew opened 4 months ago
In addition, there is no simply way to control the batch_size and the latency (https://github.com/googleapis/python-logging/blob/main/google/cloud/logging_v2/handlers/transports/background_thread.py#L133-L134). Could you add environment parameter supports to update these two configuration value?
@wguanicedew Do you know what Exception type is causing the retry function to retry immediately and not backoff?
In the stackstrace I posted in the ticket, it already said "pilot_1: google.api_core.exceptions.ResourceExhausted: 429 Quota exceeded for quota metric 'Write requests' and limit 'Write requests per minute' of service 'logging.googleapis.com' for consumer 'project_number:707963555525'. [reason: "RATE_LIMIT_EXCEEDED" "
Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
Please run down the following list and make sure you've tried the usual "quick fixes":
If you are still having issues, please be sure to include as much information as possible:
Environment details
pip --version
google-cloud-logging
version:pip show google-cloud-logging
Steps to reproduce
Code example
This part of code is problemtic https://github.com/googleapis/python-logging/blob/main/google/cloud/logging_v2/handlers/transports/background_thread.py#L129-L148.
The bckgroup_thread retry depends on the api_core/retry module. However, when the this retry function is called (https://github.com/googleapis/python-api-core/blob/main/google/api_core/retry/retry_unary.py#L153), it will raise exception immediately. So that the retry module doesn't sleep at all. At the same time, the backgroup_thread doesn't sleep either. It caused a heavy loop immediately, which seems it's attacking the logging server.
Stack trace
Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!