search

googleapis / google-cloud-cpp

C++ Client Libraries for Google Cloud Services
https://cloud.google.com/
Apache License 2.0
546 stars 365 forks source link

Fix gRPC bug: ASSERTION FAILED: initial_metadata_received_ #13724

Closed alevenberg closed 4 months ago

alevenberg commented 6 months ago

go/cloud-cxx:grpc-workaround

devbww commented 6 months ago

Add any context about the cleanup here.

It would be helpful to replace the boilerplate with something more specific.

alevenberg commented 5 months ago

Repro here: https://github.com/alevenberg/google-cloud-cpp/pull/6

coryan commented 5 months ago

I wonder if we should be waiting for this:

https://grpc.github.io/grpc/cpp/classgrpc_1_1internal_1_1_client_async_streaming_interface.html#ab5a35e289028ac2aaf062572ba6d567d

Before trying to use the metadata in one of the tracing decorators. In most RPCs that happens implicitly by the time the response arrives. For asynchronous streaming bidi RPCs (as Pub/Sub uses on the subscribe side) that may need not be the case.

alevenberg commented 5 months ago

Running: GOOGLE_CLOUD_CPP_ENABLE_TRACING=rpc,rpc-streams GOOGLE_CLOUD_CPP_ENABLE_CLOG=1 bazel run //google/cloud/pubsub/quickstart:expire

❯ bazel run //google/cloud/pubsub/quickstart:expire
INFO: Build option --test_env has changed, discarding analysis cache.
INFO: Analyzed target //google/cloud/pubsub/quickstart:expire (0 packages loaded, 6839 targets configured).
INFO: Found 1 target...
Target //google/cloud/pubsub/quickstart:expire up-to-date:
  /usr/local/google/home/alevenb/.cache/bazel/_bazel_alevenb/3e6c363550b93e009be767eef00558d7/execroot/google_cloud_cpp/bazel-out/k8-fastbuild/bin/google/cloud/pubsub/quickstart/expire
INFO: Elapsed time: 1.094s, Critical Path: 0.08s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: /usr/local/google/home/alevenb/.cache/bazel/_bazel_alevenb/3e6c363550b93e009be767eef00558d7/execroot/google_cloud_cpp/bazel-out/k8-fastbuild/bin/google/cloud/pubsub/quickstart/expire
0. Sent message with id: (10987510041345761)
Waiting for messages on expire-sub...
0. Received message with id: (10987510041345761)
0. Received message with id: (10987510041345761)
timeout reached, ending session
Received message data: "0" attributes { key: "googclient_traceparent" value: "00-2dac06d91fd0930a8bc1dd129e5625e7-0769417177a4a54f-01" } message_id: "10987510041345761" publish_time { seconds: 1713806868 nanos: 462000000 } 
E0422 13:29:12.272233420  572750 client_context.h:249]                 ASSERTION FAILED: initial_metadata_received_
[1]    571557 IOT instruction  bazel run //google/cloud/pubsub/quickstart:expire
❯ GOOGLE_CLOUD_CPP_ENABLE_TRACING=rpc,rpc-streams GOOGLE_CLOUD_CPP_ENABLE_CLOG=1 bazel run //google/cloud/pubsub/quickstart:expire
INFO: Analyzed target //google/cloud/pubsub/quickstart:expire (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //google/cloud/pubsub/quickstart:expire up-to-date:
  /usr/local/google/home/alevenb/.cache/bazel/_bazel_alevenb/3e6c363550b93e009be767eef00558d7/execroot/google_cloud_cpp/bazel-out/k8-fastbuild/bin/google/cloud/pubsub/quickstart/expire
INFO: Elapsed time: 0.350s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: /usr/local/google/home/alevenb/.cache/bazel/_bazel_alevenb/3e6c363550b93e009be767eef00558d7/execroot/google_cloud_cpp/bazel-out/k8-fastbuild/bin/google/cloud/pubsub/quickstart/expire
2024-04-22T17:29:30.055399154Z [INFO] <139907416570176> Enabled logging for gRPC calls (google/cloud/trace/v2/internal/trace_stub_factory.cc:56)
2024-04-22T17:29:30.148748539Z [WARNING] <139907416570176> MakeSubscriberConnection: Unexpected option (mangled name): N6google5cloud5v2_2426OpenTelemetryTracingOptionE (google/cloud/options.cc:33)
2024-04-22T17:29:30.182677449Z [INFO] <139907416570176> Enabled logging for gRPC calls (google/cloud/pubsub/internal/subscriber_stub_factory.cc:101)
2024-04-22T17:29:30.198345370Z [WARNING] <139907416570176> MakePublisherConnection: Unexpected option (mangled name): N6google5cloud5v2_2426OpenTelemetryTracingOptionE (google/cloud/options.cc:33)
2024-04-22T17:29:30.229516342Z [INFO] <139907416570176> Enabled logging for gRPC calls (google/cloud/pubsub/internal/publisher_stub_factory.cc:84)
2024-04-22T17:29:30.254295078Z [DEBUG] <139904801076928> AsyncPublish(1) << google.pubsub.v1.PublishRequest { topic: "projects/alevenb-test/topics/expire-topic" messages { data: "0" attributes { key: "googclient_traceparent" value: "00-c389f7b1c116a415d8c4bd01f24418c9-f45789333035a2a4-01" } } } (google/cloud/internal/log_wrapper.cc:24)
2024-04-22T17:29:30.258320985Z [DEBUG] <139904801076928> AsyncPublish(1) >> future_status=timeout (google/cloud/internal/log_wrapper.cc:37)
2024-04-22T17:29:30.439454870Z [DEBUG] <139905908848320> AsyncPublish(1) >> response=google.pubsub.v1.PublishResponse { message_ids: "10987579412149633" } (./google/cloud/internal/log_wrapper.h:76)
0. Sent message with id: (10987579412149633)
2024-04-22T17:29:30.440553919Z [DEBUG] <139907416570176> AsyncStreamingPull(2) (google/cloud/pubsub/internal/subscriber_logging_decorator.cc:121)
2024-04-22T17:29:30.440901709Z [DEBUG] <139907416570176> Start(2) << (./google/cloud/internal/async_read_write_stream_logging.h:46)
Waiting for messages on expire-sub...
2024-04-22T17:29:30.511192048Z [DEBUG] <139904750720704> Start(2) >> true (./google/cloud/internal/async_read_write_stream_logging.h:49)
2024-04-22T17:29:30.511413598Z [DEBUG] <139904750720704> Write(2) << google.pubsub.v1.StreamingPullRequest { subscription: "projects/alevenb-test/subscriptions/expire-sub" stream_ack_deadline_seconds: 10 client_id: "njbshzpmltmp5xxelxfb5omj3tfg9acv" max_outstanding_messages: 1000 max_outstanding_bytes: 104857600 } (./google/cloud/internal/async_read_write_stream_logging.h:73)
2024-04-22T17:29:30.511727727Z [DEBUG] <139904775898816> Write(2) >> true (./google/cloud/internal/async_read_write_stream_logging.h:78)
2024-04-22T17:29:30.511860967Z [DEBUG] <139904775898816> Read(2) << (./google/cloud/internal/async_read_write_stream_logging.h:57)
2024-04-22T17:29:31.421428047Z [DEBUG] <139903777167040> Read(2) >> google.pubsub.v1.StreamingPullResponse { received_messages { ack_id: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cJfQ34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." message { data: "0" attributes { key: "googclient_traceparent" value: "00-c389f7b1c116a415d8c4bd01f24418c9-f45789333035a2a4-01" } message_id: "10987579412149633" publish_time { "2024-04-22T17:29:30.429Z" } } } subscription_properties { } } (./google/cloud/internal/async_read_write_stream_logging.h:62)
2024-04-22T17:29:31.424324885Z0. Received message with id: (10987579412149633)
 [DEBUG] <139904733935296> Write(2) << google.pubsub.v1.StreamingPullRequest { } (./google/cloud/internal/async_read_write_stream_logging.h:73)
2024-04-22T17:29:31.424540635Z [DEBUG] <139904733935296> Read(2) << (./google/cloud/internal/async_read_write_stream_logging.h:57)
2024-04-22T17:29:31.424567595Z [DEBUG] <139904742328000> Write(2) >> true (./google/cloud/internal/async_read_write_stream_logging.h:78)
2024-04-22T17:29:35.059720466Z [DEBUG] <139905942419136> BatchWriteSpans() << google.devtools.cloudtrace.v2.BatchWriteSpansRequest { name: "projects/alevenb-test" spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/f93a312f8938c40e" span_id: "f93a312f8938c40e" parent_span_id: "f45789333035a2a4" display_name { value: "publisher batching" } start_time { "2024-04-22T17:29:30.229986042Z" } end_time { "2024-04-22T17:29:30.230127202Z" } attributes { attribute_map { key: "code.function" value { string_value { value: "pubsub::BatchingPublisherConnection::Publish" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } } status { } span_kind: CLIENT } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/6718983e887ddd50" span_id: "6718983e887ddd50" parent_span_id: "f45789333035a2a4" display_name { value: "publisher flow control" } start_time { "2024-04-22T17:29:30.229969202Z" } end_time { "2024-04-22T17:29:30.230143422Z" } attributes { attribute_map { key: "code.function" value { string_value { value: "pubsub::FlowControlledPublisherConnection::Publish" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } } status { } span_kind: CLIENT } spans { name: "projects/alevenb-test/traces/af81f5bfc8905c9d43f9dc5a4609b976/spans/dc92ec171a8c7c75" span_id: "dc92ec171a8c7c75" parent_span_id: "f54e0b7c6b90eded" display_name { value: "google.pubsub.v1.Publisher/Publish" } start_time { "2024-04-22T17:29:30.242902444Z" } end_time { "2024-04-22T17:29:30.43970546Z" } attributes { attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "grpc.compression_algorithm" value { string_value { value: "identity" } } } attribute_map { key: "grpc.peer" value { string_value { value: "ipv4:74.125.139.95:443" } } } attribute_map { key: "grpc.version" value { string_value { value: "1.62.1" } } } attribute_map { key: "network.transport" value { string_value { value: "ip_tcp" } } } attribute_map { key: "rpc.grpc.response.metadata.content-disposition" value { string_value { value: "attachment" } } } attribute_map { key: "rpc.grpc.response.metadata.date" value { string_value { value: "Mon, 22 Apr 2024 17:29:30 GMT" } } } attribute_map { key: "rpc.method" value { string_value { value: "Publish" } } } attribute_map { key: "rpc.service" value { string_value { value: "google.pubsub.v1.Publisher" } } } attribute_map { key: "rpc.system" value { string_value { value: "grpc" } } } } status { } span_kind: CLIENT } spans { name: "projects/alevenb-test/traces/af81f5bfc8905c9d43f9dc5a4609b976/spans/f54e0b7c6b90eded" span_id: "f54e0b7c6b90eded" display_name { value: "expire-topic publish" } start_time { "2024-04-22T17:29:30.242582654Z" } end_time { "2024-04-22T17:29:30.43982305Z" } attributes { attribute_map { key: "cloud.account.id" value { string_value { value: "google.com:cloudtop-prod" } } } attribute_map { key: "cloud.availability_zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "cloud.platform" value { string_value { value: "gcp_compute_engine" } } } attribute_map { key: "cloud.provider" value { string_value { value: "gcp" } } } attribute_map { key: "cloud.region" value { string_value { value: "us-east1" } } } attribute_map { key: "code.function" value { string_value { value: "BatchSink::AsyncPublish" } } } attribute_map { key: "g.co/r/gce_instance/instance_id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "g.co/r/gce_instance/zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "gcp.project_id" value { string_value { value: "alevenb-test" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "host.id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "host.name" value { string_value { value: "alevenb" } } } attribute_map { key: "host.type" value { string_value { value: "n2d-standard-128" } } } attribute_map { key: "messaging.batch.message_count" value { int_value: 1 } } attribute_map { key: "messaging.destination.name" value { string_value { value: "expire-topic" } } } attribute_map { key: "messaging.operation" value { string_value { value: "publish" } } } attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } attribute_map { key: "otel.scope.name" value { string_value { value: "gcloud-cpp" } } } attribute_map { key: "otel.scope.version" value { string_value { value: "v2.24.0-rc" } } } attribute_map { key: "server.address" value { string_value { value: "pubsub.googleapis.com" } } } attribute_map { key: "service.name" value { string_value { value: "unknown_service" } } } attribute_map { key: "telemetry.sdk.language" value { string_value { value: "cpp" } } } attribute_map { key: "telemetry.sdk.name" value { string_value { value: "opentelemetry" } } } attribute_map { key: "telemetry.sdk.version" value { string_value { value: "1.14.2" } } } attribute_map { key: "thread.id" value { string_value { value: "139904801076928" } } } } links { link { trace_id: "c389f7b1c116a415d8c4bd01f24418c9" span_id: "f45789333035a2a4" attributes { } } } status { } span_kind: CLIENT } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/f45789333035a2a4" span_id: "f45789333035a2a4" display_name { value: "expire-topic create" } start_time { "2024-04-22T17:29:30.229829802Z" } end_time { "2024-04-22T17:29:30.43993381Z" } attributes { attribute_map { key: "cloud.account.id" value { string_value { value: "google.com:cloudtop-prod" } } } attribute_map { key: "cloud.availability_zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "cloud.platform" value { string_value { value: "gcp_compute_engine" } } } attribute_map { key: "cloud.provider" value { string_value { value: "gcp" } } } attribute_map { key: "cloud.region" value { string_value { value: "us-east1" } } } attribute_map { key: "code.function" value { string_value { value: "pubsub::PublisherConnection::Publish" } } } attribute_map { key: "g.co/r/gce_instance/instance_id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "g.co/r/gce_instance/zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "gcp.project_id" value { string_value { value: "alevenb-test" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "host.id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "host.name" value { string_value { value: "alevenb" } } } attribute_map { key: "host.type" value { string_value { value: "n2d-standard-128" } } } attribute_map { key: "messaging.destination.name" value { string_value { value: "expire-topic" } } } attribute_map { key: "messaging.message.envelope.size" value { int_value: 21 } } attribute_map { key: "messaging.message.id" value { string_value { value: "10987579412149633" } } } attribute_map { key: "messaging.operation" value { string_value { value: "create" } } } attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } attribute_map { key: "otel.scope.name" value { string_value { value: "gcloud-cpp" } } } attribute_map { key: "otel.scope.version" value { string_value { value: "v2.24.0-rc" } } } attribute_map { key: "service.name" value { string_value { value: "unknown_service" } } } attribute_map { key: "telemetry.sdk.language" value { string_value { value: "cpp" } } } attribute_map { key: "telemetry.sdk.name" value { string_value { value: "opentelemetry" } } } attribute_map { key: "telemetry.sdk.version" value { string_value { value: "1.14.2" } } } } time_events { time_event { time { "2024-04-22T17:29:30.229995912Z" } annotation { description { value: "gl-cpp.added_to_batch" } attributes { } } } time_event { time { "2024-04-22T17:29:30.242765044Z" } annotation { description { value: "gl-cpp.publish_start" } attributes { attribute_map { key: "gcp_pubsub.publish.span_id" value { string_value { value: "f54e0b7c6b90eded" } } } attribute_map { key: "gcp_pubsub.publish.trace_id" value { string_value { value: "af81f5bfc8905c9d43f9dc5a4609b976" } } } } } } time_event { time { "2024-04-22T17:29:30.43979212Z" } annotation { description { value: "gl-cpp.publish_end" } attributes { } } } } status { } span_kind: PRODUCER } spans { name: "projects/alevenb-test/traces/ac7e76c9e831209af940d49ebc1d6aa1/spans/6b6ad6ab543d20a2" span_id: "6b6ad6ab543d20a2" display_name { value: "Start" } start_time { "2024-04-22T17:29:30.440740989Z" } end_time { "2024-04-22T17:29:30.511301428Z" } attributes { attribute_map { key: "cloud.account.id" value { string_value { value: "google.com:cloudtop-prod" } } } attribute_map { key: "cloud.availability_zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "cloud.platform" value { string_value { value: "gcp_compute_engine" } } } attribute_map { key: "cloud.provider" value { string_value { value: "gcp" } } } attribute_map { key: "cloud.region" value { string_value { value: "us-east1" } } } attribute_map { key: "g.co/r/gce_instance/instance_id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "g.co/r/gce_instance/zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "host.id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "host.name" value { string_value { value: "alevenb" } } } attribute_map { key: "host.type" value { string_value { value: "n2d-standard-128" } } } attribute_map { key: "otel.scope.name" value { string_value { value: "gcloud-cpp" } } } attribute_map { key: "otel.scope.version" value { string_value { value: "v2.24.0-rc" } } } attribute_map { key: "service.name" value { string_value { value: "unknown_service" } } } attribute_map { key: "telemetry.sdk.language" value { string_value { value: "cpp" } } } attribute_map { key: "telemetry.sdk.name" value { string_value { value: "opentelemetry" } } } attribute_map { key: "telemetry.sdk.version" value { string_value { value: "1.14.2" } } } } status { } span_kind: CLIENT } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/57d8be4cf25eb6cb" span_id: "57d8be4cf25eb6cb" parent_span_id: "cff9706dd61df18d" display_name { value: "subscriber scheduler" } start_time { "2024-04-22T17:29:31.421886616Z" } end_time { "2024-04-22T17:29:31.421892096Z" } attributes { attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } } span_kind: INTERNAL } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/8cef925e5ebd0e33" span_id: "8cef925e5ebd0e33" parent_span_id: "cff9706dd61df18d" display_name { value: "subscriber concurrency control" } start_time { "2024-04-22T17:29:31.421937246Z" } end_time { "2024-04-22T17:29:31.424259695Z" } attributes { attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } } span_kind: INTERNAL } } (google/cloud/internal/log_wrapper.cc:24)
2024-04-22T17:29:35.224653819Z [DEBUG] <139905942419136> BatchWriteSpans() >> status=OK (google/cloud/internal/log_wrapper.cc:29)
2024-04-22T17:30:12.424996629Z [DEBUG] <139904759113408> AsyncModifyAckDeadline(3) << google.pubsub.v1.ModifyAckDeadlineRequest { subscription: "projects/alevenb-test/subscriptions/expire-sub" ack_ids: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cJfQ34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." } (google/cloud/internal/log_wrapper.cc:24)
2024-04-22T17:30:12.426101108Z [DEBUG] <139904759113408> AsyncModifyAckDeadline(3) >> future_status=timeout (google/cloud/internal/log_wrapper.cc:37)
2024-04-22T17:30:12.491525981Z [DEBUG] <139903970199232> AsyncModifyAckDeadline(3) >> status=OK (google/cloud/internal/log_wrapper.cc:29)
2024-04-22T17:30:12.958867623Z [DEBUG] <139903928235712> Read(2) >> google.pubsub.v1.StreamingPullResponse { received_messages { ack_id: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cMjR34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." message { data: "0" attributes { key: "googclient_traceparent" value: "00-c389f7b1c116a415d8c4bd01f24418c9-f45789333035a2a4-01" } message_id: "10987579412149633" publish_time { "2024-04-22T17:29:30.429Z" } } } subscription_properties { } } (./google/cloud/internal/async_read_write_stream_logging.h:62)
2024-04-22T17:30:12.961637121Z [DEBUG] <139903760381632> Read(2) << (0. Received message with id: (10987579412149633)
./google/cloud/internal/async_read_write_stream_logging.h:57)
2024-04-22T17:30:15.057974822Z [DEBUG] <139905942419136> BatchWriteSpans() << google.devtools.cloudtrace.v2.BatchWriteSpansRequest { name: "projects/alevenb-test" spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/1f3c0396b3a42dd7" span_id: "1f3c0396b3a42dd7" parent_span_id: "cff9706dd61df18d" display_name { value: "expire-sub process" } start_time { "2024-04-22T17:29:31.424321275Z" } end_time { "2024-04-22T17:30:12.426214068Z" } attributes { attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } } span_kind: INTERNAL } spans { name: "projects/alevenb-test/traces/99a31e2c7ab045b91771dfce9e4e4a91/spans/bc3a7c1c09a7e29c" span_id: "bc3a7c1c09a7e29c" parent_span_id: "020ab365846740f6" display_name { value: "google.pubsub.v1.Subscriber/ModifyAckDeadline" } start_time { "2024-04-22T17:30:12.424895809Z" } end_time { "2024-04-22T17:30:12.491644831Z" } attributes { attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "grpc.compression_algorithm" value { string_value { value: "identity" } } } attribute_map { key: "grpc.peer" value { string_value { value: "ipv4:142.251.162.95:443" } } } attribute_map { key: "grpc.version" value { string_value { value: "1.62.1" } } } attribute_map { key: "network.transport" value { string_value { value: "ip_tcp" } } } attribute_map { key: "rpc.grpc.response.metadata.content-disposition" value { string_value { value: "attachment" } } } attribute_map { key: "rpc.grpc.response.metadata.date" value { string_value { value: "Mon, 22 Apr 2024 17:30:12 GMT" } } } attribute_map { key: "rpc.method" value { string_value { value: "ModifyAckDeadline" } } } attribute_map { key: "rpc.service" value { string_value { value: "google.pubsub.v1.Subscriber" } } } attribute_map { key: "rpc.system" value { string_value { value: "grpc" } } } } status { } span_kind: CLIENT } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/cff9706dd61df18d" span_id: "cff9706dd61df18d" parent_span_id: "f45789333035a2a4" display_name { value: "expire-sub subscribe" } start_time { "2024-04-22T17:29:31.421786527Z" } end_time { "2024-04-22T17:30:12.491667971Z" } attributes { attribute_map { key: "gcp.project_id" value { string_value { value: "alevenb-test" } } } attribute_map { key: "messaging.destination.name" value { string_value { value: "expire-sub" } } } attribute_map { key: "messaging.gcp_pubsub.message.ack_id" value { string_value { value: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cJfQ34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." } } } attribute_map { key: "messaging.gcp_pubsub.result" value { string_value { value: "nack" } } } attribute_map { key: "messaging.gcp_pubsub.subscription.exactly_once_delivery" value { bool_value: false } } attribute_map { key: "messaging.message.envelope.size" value { int_value: 115 } } attribute_map { key: "messaging.message.id" value { string_value { value: "10987579412149633" } } } attribute_map { key: "messaging.operation" value { string_value { value: "subscribe" } } } attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } } time_events { time_event { time { "2024-04-22T17:29:39.424387546Z" } annotation { description { value: "gl-cpp.expired" } attributes { } } } time_event { time { "2024-04-22T17:30:12.424437439Z" } annotation { description { value: "gl-cpp.message_nack" } attributes { } } } time_event { time { "2024-04-22T17:30:12.424795879Z" } annotation { description { value: "gl-cpp.nack_start" } attributes { } } } time_event { time { "2024-04-22T17:30:12.491661541Z" } annotation { description { value: "gl-cpp.nack_end" } attributes { } } } } span_kind: CONSUMER } spans { name: "projects/alevenb-test/traces/99a31e2c7ab045b91771dfce9e4e4a91/spans/020ab365846740f6" span_id: "020ab365846740f6" display_name { value: "expire-sub nack" } start_time { "2024-04-22T17:30:12.424592999Z" } end_time { "2024-04-22T17:30:12.491688311Z" } attributes { attribute_map { key: "cloud.account.id" value { string_value { value: "google.com:cloudtop-prod" } } } attribute_map { key: "cloud.availability_zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "cloud.platform" value { string_value { value: "gcp_compute_engine" } } } attribute_map { key: "cloud.provider" value { string_value { value: "gcp" } } } attribute_map { key: "cloud.region" value { string_value { value: "us-east1" } } } attribute_map { key: "code.function" value { string_value { value: "pubsub::AckHandler::nack" } } } attribute_map { key: "g.co/r/gce_instance/instance_id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "g.co/r/gce_instance/zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "gcp.project_id" value { string_value { value: "alevenb-test" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "host.id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "host.name" value { string_value { value: "alevenb" } } } attribute_map { key: "host.type" value { string_value { value: "n2d-standard-128" } } } attribute_map { key: "messaging.destination.name" value { string_value { value: "expire-sub" } } } attribute_map { key: "messaging.gcp_pubsub.message.ack_id" value { string_value { value: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cJfQ34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." } } } attribute_map { key: "messaging.gcp_pubsub.message.delivery_attempt" value { int_value: 0 } } attribute_map { key: "messaging.gcp_pubsub.subscription.template" value { string_value { value: "projects/alevenb-test/subscriptions/expire-sub" } } } attribute_map { key: "messaging.operation" value { string_value { value: "settle" } } } attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } attribute_map { key: "otel.scope.name" value { string_value { value: "gcloud-cpp" } } } attribute_map { key: "otel.scope.version" value { string_value { value: "v2.24.0-rc" } } } attribute_map { key: "service.name" value { string_value { value: "unknown_service" } } } attribute_map { key: "telemetry.sdk.language" value { string_value { value: "cpp" } } } attribute_map { key: "telemetry.sdk.name" value { string_value { value: "opentelemetry" } } } attribute_map { key: "telemetry.sdk.version" value { string_value { value: "1.14.2" } } } } links { link { trace_id: "c389f7b1c116a415d8c4bd01f24418c9" span_id: "cff9706dd61df18d" attributes { } } } status { } span_kind: INTERNAL } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/ff5498bce8bda370" span_id: "ff5498bce8bda370" parent_span_id: "480f43c39fc28f79" display_name { value: "subscriber scheduler" } start_time { "2024-04-22T17:30:12.959186023Z" } end_time { "2024-04-22T17:30:12.959189923Z" } attributes { attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } } span_kind: INTERNAL } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/a32a5037c967bac8" span_id: "a32a5037c967bac8" parent_span_id: "480f43c39fc28f79" display_name { value: "subscriber concurrency control" } start_time { "2024-04-22T17:30:12.959226853Z" } end_time { "2024-04-22T17:30:12.961605311Z" } attributes { attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } } span_kind: INTERNAL } } (google/cloud/internal/log_wrapper.cc:24)
2024-04-22T17:30:15.185336519Z [DEBUG] <139905942419136> BatchWriteSpans() >> status=OK (google/cloud/internal/log_wrapper.cc:29)
timeout reached, ending session
2024-04-22T17:30:30.442183193Z [DEBUG] <139907416570176> AsyncModifyAckDeadline(4) << google.pubsub.v1.ModifyAckDeadlineRequest { subscription: "projects/alevenb-test/subscriptions/expire-sub" ack_ids: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cMjR34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." } (google/cloud/internal/log_wrapper.cc:24)
2024-04-22T17:30:30.443143503Z [DEBUG] <139907416570176> AsyncModifyAckDeadline(4) >> future_status=timeout (google/cloud/internal/log_wrapper.cc:37)
2024-04-22T17:30:30.443214942Z [DEBUG] <139907416570176> Cancel(2) << (./google/cloud/internal/async_read_write_stream_logging.h:40)
2024-04-22T17:30:30.443855392Z [DEBUG] <139907416570176> AsyncStreamingPull(5) (google/cloud/pubsub/internal/subscriber_logging_decorator.cc:121)
2024-04-22T17:30:30.444074782Z [DEBUG] <139907416570176> Start(5) << (./google/cloud/internal/async_read_write_stream_logging.h:46)
2024-04-22T17:30:30.444591072Z [DEBUG] <139904154838720> Read(2) >> [not-set] (./google/cloud/internal/async_read_write_stream_logging.h:64)
2024-04-22T17:30:30.444906341Z [DEBUG] <139904154838720> Finish(2) << (./google/cloud/internal/async_read_write_stream_logging.h:96)
2024-04-22T17:30:30.445984351Z [DEBUG] <139903718418112> Finish(2) >> CANCELLED: CANCELLED (./google/cloud/internal/async_read_write_stream_logging.h:99)
2024-04-22T17:30:30.454449056Z [DEBUG] <139903970199232> Start(5) >> true (./google/cloud/internal/async_read_write_stream_logging.h:49)
2024-04-22T17:30:30.454678466Z [DEBUG] <139903970199232> Write(5) << google.pubsub.v1.StreamingPullRequest { subscription: "projects/alevenb-test/subscriptions/expire-sub" stream_ack_deadline_seconds: 10 client_id: "pnhv8vkiqv1rdx4l6c2nrkrzdut5c0wj" max_outstanding_messages: 1000 max_outstanding_bytes: 104857600 } (./google/cloud/internal/async_read_write_stream_logging.h:73)
2024-04-22T17:30:30.454967856Z [DEBUG] <139903886272192> Write(5) >> true (./google/cloud/internal/async_read_write_stream_logging.h:78)
2024-04-22T17:30:30.455140486Z [DEBUG] <139903886272192> Read(5) << (./google/cloud/internal/async_read_write_stream_logging.h:57)
2024-04-22T17:30:30.504061878Z [DEBUG] <139903970199232> AsyncModifyAckDeadline(4) >> status=OK (google/cloud/internal/log_wrapper.cc:29)
2024-04-22T17:30:30.975539570Z [DEBUG] <139904389834432> Read(5) >> google.pubsub.v1.StreamingPullResponse { received_messages { ack_id: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cNrR34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." message { data: "0" attributes { key: "googclient_traceparent" value: "00-c389f7b1c116a415d8c4bd01f24418c9-f45789333035a2a4-01" } message_id: "10987579412149633" publish_time { "2024-04-22T17:29:30.429Z" } } } subscription_properties { } } (./google/cloud/internal/async_read_write_stream_logging.h:62)
2024-04-22T17:30:30.978412298Z [DEBUG] <139904465368768> Write(5) << google.pubsub.v1.StreamingPullRequest { } (./google/cloud/internal/async_read_write_stream_logging.h:73)
Received message data: "0" attributes { key: "googclient_traceparent" value: "00-c389f7b1c116a415d8c4bd01f24418c9-f45789333035a2a4-01" } message_id: "10987579412149633" publish_time { seconds: 1713806970 nanos: 429000000 } 
2024-04-22T17:30:30.978761088Z [DEBUG] <139904465368768> Read(5) << (./google/cloud/internal/async_read_write_stream_logging.h:57)
2024-04-22T17:30:30.978779658Z [DEBUG] <139904540903104> Write(5) >> true (./google/cloud/internal/async_read_write_stream_logging.h:78)
2024-04-22T17:30:30.978892088Z [DEBUG] <139904482154176> AsyncAcknowledge(6) << google.pubsub.v1.AcknowledgeRequest { subscription: "projects/alevenb-test/subscriptions/expire-sub" ack_ids: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cNrR34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." } (google/cloud/internal/log_wrapper.cc:24)
2024-04-22T17:30:30.979897977Z [DEBUG] <139904482154176> AsyncAcknowledge(6) >> future_status=timeout (google/cloud/internal/log_wrapper.cc:37)
2024-04-22T17:30:31.048524219Z [DEBUG] <139904431797952> AsyncAcknowledge(6) >> status=OK (google/cloud/internal/log_wrapper.cc:29)
2024-04-22T17:30:35.061382562Z [DEBUG] <139905942419136> BatchWriteSpans() << google.devtools.cloudtrace.v2.BatchWriteSpansRequest { name: "projects/alevenb-test" spans { name: "projects/alevenb-test/traces/0eb21ba56ebf8b9cd3f2e74f5bcab15d/spans/d77fcb2e833983f3" span_id: "d77fcb2e833983f3" display_name { value: "Finish" } start_time { "2024-04-22T17:30:30.444759162Z" } end_time { "2024-04-22T17:30:30.446042391Z" } attributes { attribute_map { key: "cloud.account.id" value { string_value { value: "google.com:cloudtop-prod" } } } attribute_map { key: "cloud.availability_zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "cloud.platform" value { string_value { value: "gcp_compute_engine" } } } attribute_map { key: "cloud.provider" value { string_value { value: "gcp" } } } attribute_map { key: "cloud.region" value { string_value { value: "us-east1" } } } attribute_map { key: "g.co/r/gce_instance/instance_id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "g.co/r/gce_instance/zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "host.id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "host.name" value { string_value { value: "alevenb" } } } attribute_map { key: "host.type" value { string_value { value: "n2d-standard-128" } } } attribute_map { key: "otel.scope.name" value { string_value { value: "gcloud-cpp" } } } attribute_map { key: "otel.scope.version" value { string_value { value: "v2.24.0-rc" } } } attribute_map { key: "service.name" value { string_value { value: "unknown_service" } } } attribute_map { key: "telemetry.sdk.language" value { string_value { value: "cpp" } } } attribute_map { key: "telemetry.sdk.name" value { string_value { value: "opentelemetry" } } } attribute_map { key: "telemetry.sdk.version" value { string_value { value: "1.14.2" } } } } status { } span_kind: CLIENT } spans { name: "projects/alevenb-test/traces/956292a5aeea62cfa9a4a58fa0ba3838/spans/2af0e861fe683f1a" span_id: "2af0e861fe683f1a" display_name { value: "google.pubsub.v1.Subscriber/StreamingPull" } start_time { "2024-04-22T17:29:30.440410489Z" } end_time { "2024-04-22T17:30:30.446143001Z" } attributes { attribute_map { key: "cloud.account.id" value { string_value { value: "google.com:cloudtop-prod" } } } attribute_map { key: "cloud.availability_zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "cloud.platform" value { string_value { value: "gcp_compute_engine" } } } attribute_map { key: "cloud.provider" value { string_value { value: "gcp" } } } attribute_map { key: "cloud.region" value { string_value { value: "us-east1" } } } attribute_map { key: "g.co/r/gce_instance/instance_id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "g.co/r/gce_instance/zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 1 } } attribute_map { key: "gl-cpp.stream_started" value { bool_value: true } } attribute_map { key: "grpc.compression_algorithm" value { string_value { value: "identity" } } } attribute_map { key: "grpc.peer" value { string_value { value: "pubsub.googleapis.com" } } } attribute_map { key: "grpc.version" value { string_value { value: "1.62.1" } } } attribute_map { key: "host.id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "host.name" value { string_value { value: "alevenb" } } } attribute_map { key: "host.type" value { string_value { value: "n2d-standard-128" } } } attribute_map { key: "network.transport" value { string_value { value: "ip_tcp" } } } attribute_map { key: "otel.scope.name" value { string_value { value: "gcloud-cpp" } } } attribute_map { key: "otel.scope.version" value { string_value { value: "v2.24.0-rc" } } } attribute_map { key: "rpc.grpc.response.metadata.content-disposition" value { string_value { value: "attachment" } } } attribute_map { key: "rpc.grpc.response.metadata.date" value { string_value { value: "Mon, 22 Apr 2024 17:29:31 GMT" } } } attribute_map { key: "rpc.method" value { string_value { value: "StreamingPull" } } } attribute_map { key: "rpc.service" value { string_value { value: "google.pubsub.v1.Subscriber" } } } attribute_map { key: "rpc.system" value { string_value { value: "grpc" } } } attribute_map { key: "service.name" value { string_value { value: "unknown_service" } } } attribute_map { key: "telemetry.sdk.language" value { string_value { value: "cpp" } } } attribute_map { key: "telemetry.sdk.name" value { string_value { value: "opentelemetry" } } } attribute_map { key: "telemetry.sdk.version" value { string_value { value: "1.14.2" } } } } time_events { time_event { time { "2024-04-22T17:29:30.511335427Z" } annotation { description { value: "gl-cpp.first-write" } attributes { } } } time_event { time { "2024-04-22T17:29:30.511798977Z" } annotation { description { value: "message" } attributes { attribute_map { key: "message.id" value { int_value: 1 } } attribute_map { key: "message.is_last" value { bool_value: false } } attribute_map { key: "message.success" value { bool_value: true } } attribute_map { key: "message.type" value { string_value { value: "SENT" } } } } } } time_event { time { "2024-04-22T17:29:30.511853927Z" } annotation { description { value: "gl-cpp.first-read" } attributes { } } } time_event { time { "2024-04-22T17:29:31.421546567Z" } annotation { description { value: "message" } attributes { attribute_map { key: "message.id" value { int_value: 1 } } attribute_map { key: "message.type" value { string_value { value: "RECEIVED" } } } } } } time_event { time { "2024-04-22T17:29:31.424686405Z" } annotation { description { value: "message" } attributes { attribute_map { key: "message.id" value { int_value: 2 } } attribute_map { key: "message.is_last" value { bool_value: false } } attribute_map { key: "message.success" value { bool_value: true } } attribute_map { key: "message.type" value { string_value { value: "SENT" } } } } } } time_event { time { "2024-04-22T17:30:12.958945493Z" } annotation { description { value: "message" } attributes { attribute_map { key: "message.id" value { int_value: 2 } } attribute_map { key: "message.type" value { string_value { value: "RECEIVED" } } } } } } time_event { time { "2024-04-22T17:30:30.443205373Z" } annotation { description { value: "gl-cpp.cancel" } attributes { } } } } status { code: 2 message: "CANCELLED" } span_kind: CLIENT } spans { name: "projects/alevenb-test/traces/4379d2148dae1a46bdc3e30127c0815c/spans/9c73db78aa45d510" span_id: "9c73db78aa45d510" display_name { value: "Start" } start_time { "2024-04-22T17:30:30.443963272Z" } end_time { "2024-04-22T17:30:30.454559396Z" } attributes { attribute_map { key: "cloud.account.id" value { string_value { value: "google.com:cloudtop-prod" } } } attribute_map { key: "cloud.availability_zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "cloud.platform" value { string_value { value: "gcp_compute_engine" } } } attribute_map { key: "cloud.provider" value { string_value { value: "gcp" } } } attribute_map { key: "cloud.region" value { string_value { value: "us-east1" } } } attribute_map { key: "g.co/r/gce_instance/instance_id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "g.co/r/gce_instance/zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "host.id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "host.name" value { string_value { value: "alevenb" } } } attribute_map { key: "host.type" value { string_value { value: "n2d-standard-128" } } } attribute_map { key: "otel.scope.name" value { string_value { value: "gcloud-cpp" } } } attribute_map { key: "otel.scope.version" value { string_value { value: "v2.24.0-rc" } } } attribute_map { key: "service.name" value { string_value { value: "unknown_service" } } } attribute_map { key: "telemetry.sdk.language" value { string_value { value: "cpp" } } } attribute_map { key: "telemetry.sdk.name" value { string_value { value: "opentelemetry" } } } attribute_map { key: "telemetry.sdk.version" value { string_value { value: "1.14.2" } } } } status { } span_kind: CLIENT } spans { name: "projects/alevenb-test/traces/a145dd6168787574d8b41b51841cddc9/spans/4e7b4ab1ebcbd0db" span_id: "4e7b4ab1ebcbd0db" display_name { value: "google.pubsub.v1.Subscriber/ModifyAckDeadline" } start_time { "2024-04-22T17:30:30.441986113Z" } end_time { "2024-04-22T17:30:30.504170308Z" } attributes { attribute_map { key: "cloud.account.id" value { string_value { value: "google.com:cloudtop-prod" } } } attribute_map { key: "cloud.availability_zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "cloud.platform" value { string_value { value: "gcp_compute_engine" } } } attribute_map { key: "cloud.provider" value { string_value { value: "gcp" } } } attribute_map { key: "cloud.region" value { string_value { value: "us-east1" } } } attribute_map { key: "g.co/r/gce_instance/instance_id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "g.co/r/gce_instance/zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "grpc.compression_algorithm" value { string_value { value: "identity" } } } attribute_map { key: "grpc.peer" value { string_value { value: "ipv4:142.250.98.95:443" } } } attribute_map { key: "grpc.version" value { string_value { value: "1.62.1" } } } attribute_map { key: "host.id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "host.name" value { string_value { value: "alevenb" } } } attribute_map { key: "host.type" value { string_value { value: "n2d-standard-128" } } } attribute_map { key: "network.transport" value { string_value { value: "ip_tcp" } } } attribute_map { key: "otel.scope.name" value { string_value { value: "gcloud-cpp" } } } attribute_map { key: "otel.scope.version" value { string_value { value: "v2.24.0-rc" } } } attribute_map { key: "rpc.grpc.response.metadata.content-disposition" value { string_value { value: "attachment" } } } attribute_map { key: "rpc.grpc.response.metadata.date" value { string_value { value: "Mon, 22 Apr 2024 17:30:30 GMT" } } } attribute_map { key: "rpc.method" value { string_value { value: "ModifyAckDeadline" } } } attribute_map { key: "rpc.service" value { string_value { value: "google.pubsub.v1.Subscriber" } } } attribute_map { key: "rpc.system" value { string_value { value: "grpc" } } } attribute_map { key: "service.name" value { string_value { value: "unknown_service" } } } attribute_map { key: "telemetry.sdk.language" value { string_value { value: "cpp" } } } attribute_map { key: "telemetry.sdk.name" value { string_value { value: "opentelemetry" } } } attribute_map { key: "telemetry.sdk.version" value { string_value { value: "1.14.2" } } } } status { } span_kind: CLIENT } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/480f43c39fc28f79" span_id: "480f43c39fc28f79" parent_span_id: "f45789333035a2a4" display_name { value: "expire-sub subscribe" } start_time { "2024-04-22T17:30:12.959124273Z" } end_time { "2024-04-22T17:30:30.504196528Z" } attributes { attribute_map { key: "gcp.project_id" value { string_value { value: "alevenb-test" } } } attribute_map { key: "messaging.destination.name" value { string_value { value: "expire-sub" } } } attribute_map { key: "messaging.gcp_pubsub.message.ack_id" value { string_value { value: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cMjR34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." } } } attribute_map { key: "messaging.gcp_pubsub.result" value { string_value { value: "nack" } } } attribute_map { key: "messaging.gcp_pubsub.subscription.exactly_once_delivery" value { bool_value: false } } attribute_map { key: "messaging.message.envelope.size" value { int_value: 115 } } attribute_map { key: "messaging.message.id" value { string_value { value: "10987579412149633" } } } attribute_map { key: "messaging.operation" value { string_value { value: "subscribe" } } } attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } } time_events { time_event { time { "2024-04-22T17:30:20.96167264Z" } annotation { description { value: "gl-cpp.expired" } attributes { } } } time_event { time { "2024-04-22T17:30:30.441890933Z" } annotation { description { value: "gl-cpp.nack_start" } attributes { } } } time_event { time { "2024-04-22T17:30:30.504190018Z" } annotation { description { value: "gl-cpp.nack_end" } attributes { } } } } span_kind: CONSUMER } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/f1ae231fe14ed24d" span_id: "f1ae231fe14ed24d" parent_span_id: "83fca857308da754" display_name { value: "subscriber scheduler" } start_time { "2024-04-22T17:30:30.9758699Z" } end_time { "2024-04-22T17:30:30.97587399Z" } attributes { attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } } span_kind: INTERNAL } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/ce2d0c5973395cd1" span_id: "ce2d0c5973395cd1" parent_span_id: "83fca857308da754" display_name { value: "subscriber concurrency control" } start_time { "2024-04-22T17:30:30.97590674Z" } end_time { "2024-04-22T17:30:30.978304348Z" } attributes { attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } } span_kind: INTERNAL } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/bb3f7ab4dd5f51db" span_id: "bb3f7ab4dd5f51db" parent_span_id: "83fca857308da754" display_name { value: "expire-sub process" } start_time { "2024-04-22T17:30:30.978409269Z" } end_time { "2024-04-22T17:30:30.979986417Z" } attributes { attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } } span_kind: INTERNAL } spans { name: "projects/alevenb-test/traces/319d7873c3485af773d3e56e7e7ff51a/spans/188b148294a0e481" span_id: "188b148294a0e481" parent_span_id: "613d08394e938123" display_name { value: "google.pubsub.v1.Subscriber/Acknowledge" } start_time { "2024-04-22T17:30:30.978830408Z" } end_time { "2024-04-22T17:30:31.048645699Z" } attributes { attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "grpc.compression_algorithm" value { string_value { value: "identity" } } } attribute_map { key: "grpc.peer" value { string_value { value: "ipv4:142.251.107.95:443" } } } attribute_map { key: "grpc.version" value { string_value { value: "1.62.1" } } } attribute_map { key: "network.transport" value { string_value { value: "ip_tcp" } } } attribute_map { key: "rpc.grpc.response.metadata.content-disposition" value { string_value { value: "attachment" } } } attribute_map { key: "rpc.grpc.response.metadata.date" value { string_value { value: "Mon, 22 Apr 2024 17:30:31 GMT" } } } attribute_map { key: "rpc.method" value { string_value { value: "Acknowledge" } } } attribute_map { key: "rpc.service" value { string_value { value: "google.pubsub.v1.Subscriber" } } } attribute_map { key: "rpc.system" value { string_value { value: "grpc" } } } } status { } span_kind: CLIENT } spans { name: "projects/alevenb-test/traces/c389f7b1c116a415d8c4bd01f24418c9/spans/83fca857308da754" span_id: "83fca857308da754" parent_span_id: "f45789333035a2a4" display_name { value: "expire-sub subscribe" } start_time { "2024-04-22T17:30:30.97580319Z" } end_time { "2024-04-22T17:30:31.048688959Z" } attributes { attribute_map { key: "gcp.project_id" value { string_value { value: "alevenb-test" } } } attribute_map { key: "messaging.destination.name" value { string_value { value: "expire-sub" } } } attribute_map { key: "messaging.gcp_pubsub.message.ack_id" value { string_value { value: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cNrR34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." } } } attribute_map { key: "messaging.gcp_pubsub.result" value { string_value { value: "ack" } } } attribute_map { key: "messaging.gcp_pubsub.subscription.exactly_once_delivery" value { bool_value: false } } attribute_map { key: "messaging.message.envelope.size" value { int_value: 115 } } attribute_map { key: "messaging.message.id" value { string_value { value: "10987579412149633" } } } attribute_map { key: "messaging.operation" value { string_value { value: "subscribe" } } } attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } } time_events { time_event { time { "2024-04-22T17:30:30.978527698Z" } annotation { description { value: "gl-cpp.message_ack" } attributes { } } } time_event { time { "2024-04-22T17:30:30.978770288Z" } annotation { description { value: "gl-cpp.ack_start" } attributes { } } } time_event { time { "2024-04-22T17:30:31.048660459Z" } annotation { description { value: "gl-cpp.ack_end" } attributes { } } } } span_kind: CONSUMER } spans { name: "projects/alevenb-test/traces/319d7873c3485af773d3e56e7e7ff51a/spans/613d08394e938123" span_id: "613d08394e938123" display_name { value: "expire-sub ack" } start_time { "2024-04-22T17:30:30.978647698Z" } end_time { "2024-04-22T17:30:31.048704559Z" } attributes { attribute_map { key: "cloud.account.id" value { string_value { value: "google.com:cloudtop-prod" } } } attribute_map { key: "cloud.availability_zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "cloud.platform" value { string_value { value: "gcp_compute_engine" } } } attribute_map { key: "cloud.provider" value { string_value { value: "gcp" } } } attribute_map { key: "cloud.region" value { string_value { value: "us-east1" } } } attribute_map { key: "code.function" value { string_value { value: "pubsub::AckHandler::ack" } } } attribute_map { key: "g.co/r/gce_instance/instance_id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "g.co/r/gce_instance/zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "gcp.project_id" value { string_value { value: "alevenb-test" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "host.id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "host.name" value { string_value { value: "alevenb" } } } attribute_map { key: "host.type" value { string_value { value: "n2d-standard-128" } } } attribute_map { key: "messaging.destination.name" value { string_value { value: "expire-sub" } } } attribute_map { key: "messaging.gcp_pubsub.message.ack_id" value { string_value { value: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cNrR34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." } } } attribute_map { key: "messaging.gcp_pubsub.message.delivery_attempt" value { int_value: 0 } } attribute_map { key: "messaging.gcp_pubsub.subscription.template" value { string_value { value: "projects/alevenb-test/subscriptions/expire-sub" } } } attribute_map { key: "messaging.operation" value { string_value { value: "settle" } } } attribute_map { key: "messaging.system" value { string_value { value: "gcp_pubsub" } } } attribute_map { key: "otel.scope.name" value { string_value { value: "gcloud-cpp" } } } attribute_map { key: "otel.scope.version" value { string_value { value: "v2.24.0-rc" } } } attribute_map { key: "service.name" value { string_value { value: "unknown_service" } } } attribute_map { key: "telemetry.sdk.language" value { string_value { value: "cpp" } } } attribute_map { key: "telemetry.sdk.name" value { string_value { value: "opentelemetry" } } } attribute_map { key: "telemetry.sdk.version" value { string_value { value: "1.14.2" } } } } links { link { trace_id: "c389f7b1c116a415d8c4bd01f24418c9" span_id: "83fca857308da754" attributes { } } } status { } span_kind: INTERNAL } } (google/cloud/internal/log_wrapper.cc:24)
2024-04-22T17:30:35.201026893Z [DEBUG] <139905942419136> BatchWriteSpans() >> status=OK (google/cloud/internal/log_wrapper.cc:29)
2024-04-22T17:30:45.452477583Z [DEBUG] <139903718418112> Cancel(5) << (./google/cloud/internal/async_read_write_stream_logging.h:40)
2024-04-22T17:30:45.452999483Z [DEBUG] <139903743596224> Read(5) >> [not-set] (./google/cloud/internal/async_read_write_stream_logging.h:64)
2024-04-22T17:30:50.059723996Z [DEBUG] <139905942419136> BatchWriteSpans() << google.devtools.cloudtrace.v2.BatchWriteSpansRequest { name: "projects/alevenb-test" spans { name: "projects/alevenb-test/traces/b1a7bbc6c73382a13339ddfe871eba86/spans/b9752020d95d3829" span_id: "b9752020d95d3829" display_name { value: "google.pubsub.v1.Subscriber/StreamingPull" } start_time { "2024-04-22T17:30:30.443749262Z" } end_time { "2024-04-22T17:30:45.453210923Z" } attributes { attribute_map { key: "cloud.account.id" value { string_value { value: "google.com:cloudtop-prod" } } } attribute_map { key: "cloud.availability_zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "cloud.platform" value { string_value { value: "gcp_compute_engine" } } } attribute_map { key: "cloud.provider" value { string_value { value: "gcp" } } } attribute_map { key: "cloud.region" value { string_value { value: "us-east1" } } } attribute_map { key: "g.co/r/gce_instance/instance_id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "g.co/r/gce_instance/zone" value { string_value { value: "us-east1-b" } } } attribute_map { key: "gl-cpp.status_code" value { int_value: 0 } } attribute_map { key: "gl-cpp.stream_started" value { bool_value: true } } attribute_map { key: "grpc.compression_algorithm" value { string_value { value: "identity" } } } attribute_map { key: "grpc.peer" value { string_value { value: "pubsub.googleapis.com" } } } attribute_map { key: "grpc.version" value { string_value { value: "1.62.1" } } } attribute_map { key: "host.id" value { string_value { value: "7828862928603536709" } } } attribute_map { key: "host.name" value { string_value { value: "alevenb" } } } attribute_map { key: "host.type" value { string_value { value: "n2d-standard-128" } } } attribute_map { key: "network.transport" value { string_value { value: "ip_tcp" } } } attribute_map { key: "otel.scope.name" value { string_value { value: "gcloud-cpp" } } } attribute_map { key: "otel.scope.version" value { string_value { value: "v2.24.0-rc" } } } attribute_map { key: "rpc.grpc.response.metadata.content-disposition" value { string_value { value: "attachment" } } } attribute_map { key: "rpc.grpc.response.metadata.date" value { string_value { value: "Mon, 22 Apr 2024 17:30:30 GMT" } } } attribute_map { key: "rpc.method" value { string_value { value: "StreamingPull" } } } attribute_map { key: "rpc.service" value { string_value { value: "google.pubsub.v1.Subscriber" } } } attribute_map { key: "rpc.system" value { string_value { value: "grpc" } } } attribute_map { key: "service.name" value { string_value { value: "unknown_service" } } } attribute_map { key: "telemetry.sdk.language" value { string_value { value: "cpp" } } } attribute_map { key: "telemetry.sdk.name" value { string_value { value: "opentelemetry" } } } attribute_map { key: "telemetry.sdk.version" value { string_value { value: "1.14.2" } } } } time_events { time_event { time { "2024-04-22T17:30:30.454587576Z" } annotation { description { value: "gl-cpp.first-write" } attributes { } } } time_event { time { "2024-04-22T17:30:30.455074906Z" } annotation { description { value: "message" } attributes { attribute_map { key: "message.id" value { int_value: 1 } } attribute_map { key: "message.is_last" value { bool_value: false } } attribute_map { key: "message.success" value { bool_value: true } } attribute_map { key: "message.type" value { string_value { value: "SENT" } } } } } } time_event { time { "2024-04-22T17:30:30.455133276Z" } annotation { description { value: "gl-cpp.first-read" } attributes { } } } time_event { time { "2024-04-22T17:30:30.9756218Z" } annotation { description { value: "message" } attributes { attribute_map { key: "message.id" value { int_value: 1 } } attribute_map { key: "message.type" value { string_value { value: "RECEIVED" } } } } } } time_event { time { "2024-04-22T17:30:30.978912338Z" } annotation { description { value: "message" } attributes { attribute_map { key: "message.id" value { int_value: 2 } } attribute_map { key: "message.is_last" value { bool_value: false } } attribute_map { key: "message.success" value { bool_value: true } } attribute_map { key: "message.type" value { string_value { value: "SENT" } } } } } } time_event { time { "2024-04-22T17:30:45.452449183Z" } annotation { description { value: "gl-cpp.cancel" } attributes { } } } } status { } span_kind: CLIENT } } (google/cloud/internal/log_wrapper.cc:24)
2024-04-22T17:30:50.276678653Z [DEBUG] <139905942419136> BatchWriteSpans() >> status=OK (google/cloud/internal/log_wrapper.cc:29)
2024-04-22T17:30:53.962336088Z [DEBUG] <139904524117696> AsyncModifyAckDeadline(7) << google.pubsub.v1.ModifyAckDeadlineRequest { subscription: "projects/alevenb-test/subscriptions/expire-sub" ack_ids: "PkVTRFAGFixdRkhRNxkIaFEOT14jPzUgKEUQAAZneDsMRhkLaTNcdQdRDRh6K2FzPggTVAFMWX5VWzM9cMjR34ZnLzhzfWh9bV8UCAZEUX9bUh0Oa3TJxuD9tq35XntJ...<truncated>..." } (google/cloud/internal/log_wrapper.cc:24)
2024-04-22T17:30:53.962503788Z [DEBUG] <139904524117696> AsyncModifyAckDeadline(7) >> future_status=ready (google/cloud/internal/log_wrapper.cc:37)
2024-04-22T17:30:53.962576908Z [DEBUG] <139904524117696> AsyncModifyAckDeadline(7) >> status=CANCELLED: call cancelled (google/cloud/internal/log_wrapper.cc:29)
E0422 13:30:53.962689928  574959 client_context.h:249]                 ASSERTION FAILED: initial_metadata_received_
[1]    573772 IOT instruction  GOOGLE_CLOUD_CPP_ENABLE_TRACING=rpc,rpc-streams GOOGLE_CLOUD_CPP_ENABLE_CLOG=
alevenberg commented 5 months ago

It occurs right after the modack is cancelled at the end of the stream

2024-04-22T17:30:53.962576908Z [DEBUG] <139904524117696> AsyncModifyAckDeadline(7) >> status=CANCELLED: call cancelled (google/cloud/internal/log_wrapper.cc:29)
alevenberg commented 5 months ago

I think @coryan's link is correct. We want to check ReadInitialMetadata before calling GetServerInitialMetadata

Our current stub interface does not include a AsyncStreamingInterface, but only the grpc::ClientContex

alevenberg commented 5 months ago

The protoc generated stub has

    virtual ::grpc::ClientAsyncResponseReaderInterface< ::google::protobuf::Empty>* AsyncModifyAckDeadlineRaw(::grpc::ClientContext* context, const ::google::pubsub::v1::ModifyAckDeadlineRequest& request, ::grpc::CompletionQueue* cq) = 0;
    std::unique_ptr< ::grpc::ClientAsyncResponseReader< ::google::protobuf::Empty>> AsyncModifyAckDeadline(::grpc::ClientContext* context, const ::google::pubsub::v1::ModifyAckDeadlineRequest& request, ::grpc::CompletionQueue* cq) {
      return std::unique_ptr< ::grpc::ClientAsyncResponseReader< ::google::protobuf::Empty>>(AsyncModifyAckDeadlineRaw(context, request, cq));
    }

But that returns it after we make the call, and we try and access the InitialMetadata before

alevenberg commented 5 months ago 
Completion will be notified by tag on the associated completion queue. This call is optional, but if it is used, it cannot be used concurrently with or after the [AsyncReaderInterface::Read](https://grpc.github.io/grpc/cpp/classgrpc_1_1internal_1_1_async_reader_interface.html#adf87c602036d69158950b7299d0aae70) method.
alevenberg commented 5 months ago 
  /// Prepare the operation to receive the response and start the RPC.
  template <typename AsyncFunctionType>
  void Start(AsyncFunctionType async_call,
             // NOLINTNEXTLINE(performance-unnecessary-value-param)
             std::shared_ptr<grpc::ClientContext> context,
             Request const& request, grpc::CompletionQueue* cq, void* tag) {
    promise_ = promise<StatusOr<Response>>([context] { context->TryCancel(); });
    auto rpc = async_call(context.get(), request, cq);
    void* intial_metadata_tag;
    bool inital_metadata_ok = false;
    auto deadline = std::chrono::system_clock::now() + std::chrono::seconds(30);
    cq->AsyncNext(&intial_metadata_tag, &inital_metadata_ok, deadline);
    if (inital_metadata_ok) {
      std::cout << "asking for metadata\n";
      rpc->ReadInitialMetadata(intial_metadata_tag);
      bool ok = true;
      if (!cq->Next(&intial_metadata_tag, &ok) || !ok) {
        std::cout << "operation failed.\n";
      }
    }
    rpc->Finish(&response_, &status_, tag);
  }

async_rpc_details.h

This stopped the unary rpc error. We are getting an error now after the AsyncStreamingPull call.

2024-04-22T19:49:57.236206238Z [DEBUG] <140681116022464> AsyncStreamingPull(10) (google/cloud/pubsub/internal/subscriber_logging_decorator.cc:121)
2024-04-22T19:49:57.236500908Z [DEBUG] <140681116022464> Start(10) << (./google/cloud/internal/async_read_write_stream_logging.h:46)
2024-04-22T19:49:57.236577438Z [DEBUG] <140681116022464> Start(10) >> false (./google/cloud/internal/async_read_write_stream_logging.h:49)
2024-04-22T19:49:57.236791238Z [DEBUG] <140681116022464> Finish(10) << (./google/cloud/internal/async_read_write_stream_logging.h:96)
2024-04-22T19:49:57.236873778Z [DEBUG] <140681116022464> Finish(10) >> OK (./google/cloud/internal/async_read_write_stream_logging.h:99)
2024-04-22T19:49:57.236961758Z [INFO] <140681116022464> alevenb - GetServerInitialMetadata
E0422 15:49:57.237017698  667598 client_context.h:249]                 ASSERTION FAILED: initial_metadata_received_
[1]    664899 IOT instruction  GOOGLE_CLOUD_CPP_ENABLE_TRACING=rpc,rpc-streams GOOGLE_CLOUD_CPP_ENABLE_CLOG=
alevenberg commented 5 months ago

Note: darren also told me this is triggered in the EndSpan when we call GetAttributes(). So it might be worth trying to see what span is generated and triggering in the error.

alevenberg commented 5 months ago

I added the above to the Streaming read and that fixed the grpc assertion error.

I will now add this information to the document

alevenberg commented 4 months ago

I tried applying the following patch

diff --git a/google/cloud/internal/grpc_opentelemetry.h b/google/cloud/internal/grpc_opentelemetry.h
index 929b6935fb..c64be3154e 100644
--- a/google/cloud/internal/grpc_opentelemetry.h
+++ b/google/cloud/internal/grpc_opentelemetry.h
@@ -18,6 +18,7 @@
 #include "google/cloud/completion_queue.h"
 #include "google/cloud/internal/grpc_metadata_view.h"
 #include "google/cloud/internal/opentelemetry.h"
+#include "google/cloud/internal/status_utils.h"
 #include "google/cloud/options.h"
 #include "google/cloud/version.h"
 #include <grpcpp/grpcpp.h>
@@ -86,6 +87,22 @@ T EndSpan(grpc::ClientContext& context, opentelemetry::trace::Span& span,
   return EndSpan(span, std::move(value));
 }
 template <typename T>
 future<T> EndSpan(
     std::shared_ptr<grpc::ClientContext> context,
@@ -94,7 +111,10 @@ future<T> EndSpan(
   return fut.then([oc = opentelemetry::context::RuntimeContext::GetCurrent(),
                    c = std::move(context), s = std::move(span)](auto f) {
     auto t = f.get();
-    ExtractAttributes(*c, *s, GrpcMetadataView::kWithServerMetadata);
+    ExtractAttributes(*c, *s,
+                      IsClientOrigin(t.status())
+                          ? GrpcMetadataView::kWithoutServerMetadata
+                          : GrpcMetadataView::kWithServerMetadata);
     DetachOTelContext(oc);
     return EndSpan(*s, std::move(t));
   });

Then ran 

bazel test google/cloud:internal_grpc_opentelemetry_test

and got

google/cloud/internal/grpc_opentelemetry_test.cc:149:19:   required from here
./google/cloud/internal/grpc_opentelemetry.h:115:37: error: 'class google::cloud::v2_25::Status' has no member named 'status'; did you mean 'Status'?
  115 |                       IsClientOrigin(t.status())
      |                       ~~~~~~~~~~~~~~^~~~~~~~~~~~

Because sometimes the T matches with Status, not StatusOr.

So I changed the patch to

diff --git a/google/cloud/internal/grpc_opentelemetry.h b/google/cloud/internal/grpc_opentelemetry.h
index 929b6935fb..2990cf3083 100644
--- a/google/cloud/internal/grpc_opentelemetry.h
+++ b/google/cloud/internal/grpc_opentelemetry.h
@@ -18,6 +18,7 @@
 #include "google/cloud/completion_queue.h"
 #include "google/cloud/internal/grpc_metadata_view.h"
 #include "google/cloud/internal/opentelemetry.h"
+#include "google/cloud/internal/status_utils.h"
 #include "google/cloud/options.h"
 #include "google/cloud/version.h"
 #include <grpcpp/grpcpp.h>
@@ -86,15 +87,34 @@ T EndSpan(grpc::ClientContext& context, opentelemetry::trace::Span& span,
   return EndSpan(span, std::move(value));
 }

+future<Status> EndSpan(
+    std::shared_ptr<grpc::ClientContext> context,
+    opentelemetry::nostd::shared_ptr<opentelemetry::trace::Span> span,
+    future<Status> fut) {
+  return fut.then([oc = opentelemetry::context::RuntimeContext::GetCurrent(),
+                   c = std::move(context), s = std::move(span)](auto f) {
+    auto t = f.get();
+    ExtractAttributes(*c, *s,
+                      IsClientOrigin(t)
+                          ? GrpcMetadataView::kWithoutServerMetadata
+                          : GrpcMetadataView::kWithServerMetadata);
+    DetachOTelContext(oc);
+    return EndSpan(*s, std::move(t));
+  });
+}
+
 template <typename T>
-future<T> EndSpan(
+future<StatusOr<T>> EndSpan(
     std::shared_ptr<grpc::ClientContext> context,
     opentelemetry::nostd::shared_ptr<opentelemetry::trace::Span> span,
-    future<T> fut) {
+    future<StatusOr<T>> fut) {
   return fut.then([oc = opentelemetry::context::RuntimeContext::GetCurrent(),
                    c = std::move(context), s = std::move(span)](auto f) {
     auto t = f.get();
-    ExtractAttributes(*c, *s, GrpcMetadataView::kWithServerMetadata);
+    ExtractAttributes(*c, *s,
+                      IsClientOrigin(t.status())
+                          ? GrpcMetadataView::kWithoutServerMetadata
+                          : GrpcMetadataView::kWithServerMetadata);
     DetachOTelContext(oc);
     return EndSpan(*s, std::move(t));
   });

And got 

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
/usr/bin/ld.gold: error: bazel-out/k8-fastbuild/bin/google/cloud/_objs/google_cloud_cpp_grpc_utils_private/grpc_opentelemetry.pic.o: multiple definition of 'google::cloud::v2_25::internal::EndSpan(std::shared_ptr<grpc::ClientContext>, std::shared_ptr<opentelemetry::v1::trace::Span>, google::cloud::v2_25::future<google::cloud::v2_25::Status>)'
/usr/bin/ld.gold: bazel-out/k8-fastbuild/bin/google/cloud/_objs/google_cloud_cpp_grpc_utils_private/async_polling_loop.pic.o: previous definition here
/usr/bin/ld.gold: error: bazel-out/k8-fastbuild/bin/google/cloud/_objs/google_cloud_cpp_grpc_utils_private/minimal_iam_credentials_stub.pic.o: multiple definition of 'google::cloud::v2_25::internal::EndSpan(std::shared_ptr<grpc::ClientContext>, std::shared_ptr<opentelemetry::v1::trace::Span>, google::cloud::v2_25::future<google::cloud::v2_25::Status>)'
/usr/bin/ld.gold: bazel-out/k8-fastbuild/bin/google/cloud/_objs/google_cloud_cpp_grpc_utils_private/async_polling_loop.pic.o: previous definition here
collect2: error: ld returned 1 exit status

Can someone who knows more about templates help me here?