Help wanted for @opentelemetry/instrumentation-http: TypeError: Invalid value "undefined" for header "authorization"

[x] This only affects the JavaScript OpenTelemetry library

I'm looking for help/advice on tracking down an error when enabling @opentelemetry/instrumentation-http in a node app via @opentelemetry/auto-instrumentations-node

I'm a node ignoramus, but I'm hoping that even if it's a stupid question it'll help someone else later. When enabling tracing with --require tracing.ts and either of the typescript examples I include at the end of this ticket, I see errors like the following when my Express app starts up:

[2] TypeError: Invalid value "undefined" for header "authorization"
[2]     at ClientRequest.setHeader (node:_http_outgoing:576:3)
[2]     at new ClientRequest (node:_http_client:256:14)
[2]     at request (node:https:353:10)
[2]     at server/node_modules/@opentelemetry/instrumentation-http/src/http.ts:583:26
[2]     at Object.safeExecuteInTheMiddle (server/node_modules/@opentelemetry/instrumentation/src/utils.ts:32:14)
[2]     at server/node_modules/@opentelemetry/instrumentation-http/src/http.ts:582:45
[2]     at AsyncLocalStorage.run (node:async_hooks:320:14)
[2]     at AsyncLocalStorageContextManager.with (server/node_modules/@opentelemetry/context-async-hooks/src/AsyncLocalStorageContextManager.ts:40:36)
[2]     at ContextAPI.with (server/node_modules/@opentelemetry/api/src/api/context.ts:77:42)
[2]     at outgoingRequest (server/node_modules/@opentelemetry/instrumentation-http/src/http.ts:572:26)
[2] [ERROR] 22:10:53 TypeError: Invalid value "undefined" for header "authorization"

I can't find any mention of an authorization header anywhere in the app codebase, nor any relevant looking ones in opentelemetry-js or opentelemetry-js-contrib. And I can't for the life of me figure out how the stack trace relates to the final error.

It seems to hit variants of the same stack trace depending on whichever outbound http request it makes first, no matter what the target endpoint/service is.

I've tried using only the console span exporter, disabling zipkin/otel/prom exporters. No change.

I'm invoking the app via umi dev --require ./tracing.ts in package.json in case the umi framework is relevant.

Versions:

node v16.9.1
npm 7.21.1

and from package.json:

    "@umijs/fabric": "2.10.1",
    @opentelemetry/api: ^1.1.0,    
    @opentelemetry/auto-instrumentations-node: ^0.28.0,    
    @opentelemetry/exporter-collector: ^0.25.0,    
    @opentelemetry/exporter-prometheus: ^0.27.0,    
    @opentelemetry/exporter-trace-otlp-http: ^0.27.0,    
    @opentelemetry/exporter-zipkin: ^1.1.1,    
    @opentelemetry/sdk-node: ^0.27.0,

Interestingly I don't see a similar error from the React app loading the same instrumentation wrapper, but I'm not sure if it's just not hitting the instrumented code yet. See: Node ignoramus trying to instrument an app that isn't mine.

Instrumentation `tracing.ts` version 1

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { CollectorTraceExporter } = require('@opentelemetry/exporter-collector');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');

const exporter = new CollectorTraceExporter();
const provider = new NodeTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'basic-service',
  }),
});
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

registerInstrumentations({
  instrumentations: [
    getNodeAutoInstrumentations({
      // load custom configuration for http instrumentation
      '@opentelemetry/instrumentation-http': {
        applyCustomAttributesOnSpan: (span) => {
          span.setAttribute('foo2', 'bar2');
        },
      },
    }),
  ],
});

Instrumentation `tracing.ts` version 2

const opentelemetry = require("@opentelemetry/sdk-node");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const sdk = new opentelemetry.NodeSDK({
  traceExporter: new opentelemetry.tracing.ConsoleSpanExporter(),
  instrumentations: [getNodeAutoInstrumentations()]
});
sdk.start().then(() => {
  // Resources have been detected and SDK is started
});

open-telemetry / opentelemetry-js