search

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.85k stars 4.25k forks source link

[Feature Request]: Lower log-level for caught error in StorageApiWritesShardedRecords #22537

Open TheNeuralBit opened 2 years ago

TheNeuralBit commented 2 years ago

What would you like to happen?

We log an error here: https://github.com/apache/beam/blob/e5e3cb25ca4fc2e31c10eb3dbda8289c6bfc7140/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWritesShardedRecords.java#L447-L448

But I've seen that these errors can be benign, e.g. ALREADY_EXISTS: The offset is within stream, expected offset ... can happen when re-establishing a stream, but doesn't indicate a failure, so logging it as an error is misleading.

Issue Priority

Priority: 2

Issue Component

Component: io-java-gcp

TheNeuralBit commented 2 years ago

@reuvenlax should we lower the log level here? I'm not sure if there are other cases where logging an error is appropriate, but it seems like at least in the case I noted it's not.