Log GCS upload ID for Python GCS connector

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.

https://beam.apache.org/

Apache License 2.0

7.85k stars 4.25k forks source link

Log GCS upload ID for Python GCS connector #21577

Open damccorm opened 2 years ago

damccorm commented 2 years ago

We already do this for Java: https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/UploadIdResponseInterceptor.java

To prevent excessive logging, this should only be done when DEBUG logs are enabled.

Imported from Jira BEAM-14147. Original Jira may contain additional context. Reported by: chamikara.

Geddy05 commented 1 year ago

.take-issue

liferoad commented 1 year ago

@Geddy05 are you still working on this issue?

yuwtennis commented 7 months ago

Hello. May I ask if this issue is already implemented ?
I see upload_id when I increase the log level in v2.54.0.

DEBUG:urllib3.connectionpool:https://storage.googleapis.com:443 "PUT /upload/storage/v1/b/MY_PROJECT/o?uploadType=resumable&upload_id=ABPtcPqDgGcN_cmYiKCbbu4McvSfJmQx64VuFR_sQPHXbZVCE_RPBNbqJSQVqju83uev-Skbo6--j25ctUSDhHeyx
7h7dyTntFT9fHZKhN93wvuL HTTP/1.1" 200 1164