Should a rateLimitExceeded have a "429 Too Many Requests" instead of "403 Forbidden"?

cozos commented 1 month ago

In many cases, users want to be able to retry rateLimitExceeded errors. Many retry libraries and codebases support this by automatically retrying HTTP exceptions with a 429 Too Many Requests HTTP error, and any 5XX error codes. For example, BigQuery itself. However BigQuery will raise a 403 Forbidden for rate limit problems - see: https://github.com/googleapis/python-bigquery/blob/f55864ec3d6381f2b31598428a64822fdc73cb56/google/cloud/bigquery/job/base.py#L49

A 403 Forbidden implies that the API user does not have the permissions or authentication credentials to do the thing. That seems inappropriate for a rateLimitError, right?

This mis-coding has two consequences for retry logic:

In codebases where you'd expect to get retries for free, you don't, because libraries won't retry a 403 Forbidden, and rightly so - why would you retry an access error.
Users who do want to implement retries for BQ rateLimitExceeded errors now have to retry on 403 Forbidden, possibly causing them to retry genuine permissions/auth errors that will fail no matter how many attempts.

ohmayr commented 2 weeks ago

Transferring this issue to the Bigquery team since the error is raised from the backend API and not the client.

chalmerlowe commented 2 weeks ago

This should be fixed by this PR. It seems like this might be a duplicate of Issue #1985

https://github.com/googleapis/python-bigquery/pull/1994

Closing.

googleapis / python-bigquery

Should a rateLimitExceeded have a "429 Too Many Requests" instead of "403 Forbidden"? #2006