BigQuery Storage Write API v1: Request 'AppendRows' from role 'cloud-dataengine-globalrouting' throttled: Task is overloaded (memory-protection) go/tr-t

[etrandafir93]

Environment details

Java 21 Spring Boot 3.3.3 google-cloud-bigquery: 2.42.1 google-cloud-bigquerystorage: 3.9.0

Steps to reproduce

1. Listen to a Kafka topic
2. For each message, send one or more records (within a JSONArray) to BigQuery, using a JsonStreamWriter
3. If sending to BQ fails, retry up to 9 times with a delay of one second

Code example

The code is pretty straight-forward, nothing special here:

private final JsonStreamWriter streamWriter;

// retried up to 9 times with a 1s delay
public AppendRowsResponse write(JSONArray json) {
      var resp = streamWriter.append(json).get();
     // ...
}

Stack trace

java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.cloud.bigquery.storage.v1.Exceptions$StreamWriterClosedException: FAILED_PRECONDITION: Connection is closed due to com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Request 'AppendRows' from role 'cloud-dataengine-globalrouting' throttled: Task is overloaded (memory-protection) go/tr-t.
    [ ... ]
Caused by: java.util.concurrent.ExecutionException: com.google.cloud.bigquery.storage.v1.Exceptions$StreamWriterClosedException: FAILED_PRECONDITION: Connection is closed due to com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Request 'AppendRows' from role 'cloud-dataengine-globalrouting' throttled: Task is overloaded (memory-protection) go/tr-t.
    at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:595)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:554)
    at com.google.api.core.AbstractApiFuture.get(AbstractApiFuture.java:53)
    at com.betvictor.mktp.bigquery.Writer.write(Writer.java:40)
    ... 11 common frames omitted
Caused by: com.google.cloud.bigquery.storage.v1.Exceptions$StreamWriterClosedException: FAILED_PRECONDITION: Connection is closed due to com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Request 'AppendRows' from role 'cloud-dataengine-globalrouting' throttled: Task is overloaded (memory-protection) go/tr-t.
    at com.google.cloud.bigquery.storage.v1.ConnectionWorker.appendInternal(ConnectionWorker.java:586)
    at com.google.cloud.bigquery.storage.v1.ConnectionWorker.append(ConnectionWorker.java:529)
    at com.google.cloud.bigquery.storage.v1.StreamWriter$SingleConnectionOrConnectionPool.append(StreamWriter.java:173)
    at com.google.cloud.bigquery.storage.v1.StreamWriter.appendWithUniqueId(StreamWriter.java:503)
    at com.google.cloud.bigquery.storage.v1.SchemaAwareStreamWriter.appendWithUniqueId(SchemaAwareStreamWriter.java:258)
    at com.google.cloud.bigquery.storage.v1.SchemaAwareStreamWriter.append(SchemaAwareStreamWriter.java:139)
    at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:65)
    ... 12 common frames omitted

Any additional information below

I've noticed similar issues being reported for the nodejs and dotnet clients.

[leahecole]

I know this involves both BQ and BQ Storage, but I transferred it to the BQ storage repo because I think it skews slightly storagey-er, and I'm assigning to my java colleagues to address. Thanks for linking to the node and dotnet clients!

[Priy98748034]

com.google.apis google-api-services-androidmanagement v1-rev20241022-2.0.0

[etrandafir93]

hello, I have managed to sort out the issue. This application was writing to different BigQuery tables and it seems that a different module (writing to a different table) was using too many connections, causing this throttling issue.

After starting to use multiplexing - and using a shared connection pool - the issue was solved.