search

googleapis / java-bigquerystorage

Apache License 2.0
60 stars 80 forks source link

BigQuery Storage Write API v1: Request 'AppendRows' from role 'cloud-dataengine-globalrouting' throttled: Task is overloaded (memory-protection) go/tr-t #2725

Open etrandafir93 opened 1 week ago

etrandafir93 commented 1 week ago

Environment details

Java 21 Spring Boot 3.3.3 google-cloud-bigquery: 2.42.1 google-cloud-bigquerystorage: 3.9.0

Steps to reproduce

  1. Listen to a Kafka topic
  2. For each message, send one or more records (within a JSONArray) to BigQuery, using a JsonStreamWriter
  3. If sending to BQ fails, retry up to 9 times with a delay of one second

Code example

The code is pretty straight-forward, nothing special here: 

private final JsonStreamWriter streamWriter;

// retried up to 9 times with a 1s delay
public AppendRowsResponse write(JSONArray json) {
      var resp = streamWriter.append(json).get();
     // ...
}

Stack trace

java.lang.RuntimeException: java.util.concurrent.ExecutionException: com.google.cloud.bigquery.storage.v1.Exceptions$StreamWriterClosedException: FAILED_PRECONDITION: Connection is closed due to com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Request 'AppendRows' from role 'cloud-dataengine-globalrouting' throttled: Task is overloaded (memory-protection) go/tr-t.
    [ ... ]
Caused by: java.util.concurrent.ExecutionException: com.google.cloud.bigquery.storage.v1.Exceptions$StreamWriterClosedException: FAILED_PRECONDITION: Connection is closed due to com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Request 'AppendRows' from role 'cloud-dataengine-globalrouting' throttled: Task is overloaded (memory-protection) go/tr-t.
    at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:595)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:554)
    at com.google.api.core.AbstractApiFuture.get(AbstractApiFuture.java:53)
    at com.betvictor.mktp.bigquery.Writer.write(Writer.java:40)
    ... 11 common frames omitted
Caused by: com.google.cloud.bigquery.storage.v1.Exceptions$StreamWriterClosedException: FAILED_PRECONDITION: Connection is closed due to com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: Request 'AppendRows' from role 'cloud-dataengine-globalrouting' throttled: Task is overloaded (memory-protection) go/tr-t.
    at com.google.cloud.bigquery.storage.v1.ConnectionWorker.appendInternal(ConnectionWorker.java:586)
    at com.google.cloud.bigquery.storage.v1.ConnectionWorker.append(ConnectionWorker.java:529)
    at com.google.cloud.bigquery.storage.v1.StreamWriter$SingleConnectionOrConnectionPool.append(StreamWriter.java:173)
    at com.google.cloud.bigquery.storage.v1.StreamWriter.appendWithUniqueId(StreamWriter.java:503)
    at com.google.cloud.bigquery.storage.v1.SchemaAwareStreamWriter.appendWithUniqueId(SchemaAwareStreamWriter.java:258)
    at com.google.cloud.bigquery.storage.v1.SchemaAwareStreamWriter.append(SchemaAwareStreamWriter.java:139)
    at com.google.cloud.bigquery.storage.v1.JsonStreamWriter.append(JsonStreamWriter.java:65)
    ... 12 common frames omitted

Any additional information below

I've noticed similar issues being reported for the nodejs and dotnet clients.

leahecole commented 1 week ago

I know this involves both BQ and BQ Storage, but I transferred it to the BQ storage repo because I think it skews slightly storagey-er, and I'm assigning to my java colleagues to address. Thanks for linking to the node and dotnet clients!