spanner: out of range [0] with length 0 when OTMetrics enabled after client creation

[tamayika]

Client

Spanner

Environment

N/A

Go Environment

N/A

Code

e.g.

package main

func main() {
    client, err := spanner.NewClientWithConfig(context.Background(), "", spanner.ClientConfig{
        OpenTelemetryMeterProvider: metric.NewMeterProvider(),
    })
    if err != nil {
        panic(err)
    }
    spanner.EnableOpenTelemetryMetrics()
    err = client.ReadOnlyTransaction().Query(context.Background(), spanner.NewStatement("SELECT 1"))
    if err != nil {
        panic(err)
    }
}

Expected behavior

No panic

Actual behavior

panic with out of range [0] with length 0 at https://github.com/googleapis/google-cloud-go/blob/main/spanner/ot_metrics.go#L245 . Screenshots

N/A

Additional context

I'm using Spanner Emulator, so there is no server-timing metadata in the response. And client.otConfig is always empty before EnableOpenTelemetryMetrics called. Our codebase is modular monolith, so some service creates client without EnableOpenTelemetryMetrics but other service creates client with EnableOpenTelemetryMetrics and panics.

The code should be

    if len(md.Get("server-timing")) == 0 {
        if otConfig.gfeHeaderMissingCount != nil {
            otConfig.gfeHeaderMissingCount.Add(ctx, 1, metric.WithAttributes(attr...))
        }
        return nil
    }

[rahul2393]

Related ticket https://github.com/googleapis/google-cloud-go/issues/9519, spanner.EnableOpenTelemetryMetrics() enable OpenTelemetry for all the clients in application and should be called before any client initialization, panic here is not expected though which I am working to fix here https://github.com/googleapis/google-cloud-go/pull/9657, but for short term please move spanner.EnableOpenTelemetryMetrics() before spanner.NewClientWithConfig

[tamayika]

I think several options should exists to use otel metrics.

• spanner.EnableOpenTelemetryMetrics() activates all client metrics. ClientConfig.OpenTelemetryMeterProvider is optional.(i.e. use global meter provider if nil)
• When ClientConfig.OpenTelemetryMeterProvider is set but spanner.EnableOpenTelemetryMetrics() is not called, metrics is activated only for its client.