search

open-telemetry / opentelemetry-python-contrib

OpenTelemetry instrumentation for Python modules
https://opentelemetry.io
Apache License 2.0
734 stars 606 forks source link

Latest python auto-instrumentation doesn't work - Requested component 'otlp_proto_grpc' not found in entry points for 'opentelemetry_metrics_exporter' #1361

Closed dominthomas closed 2 years ago

dominthomas commented 2 years ago

Describe your environment

Additional Info:

Steps to reproduce Following the official guide from here: https://github.com/open-telemetry/opentelemetry-operator#opentelemetry-auto-instrumentation-injection

What is the expected behavior? OpenTelemetry being initialised & traces befing forwarded

What is the actual behavior? The following error is raised after the init container performs the auto-injection.

Configuration of configurator failed
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 105, in _load_configurators
    entry_point.load()().configure(auto_instrumentation_version=__version__)  # type: ignore
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 273, in configure
    self._configure(**kwargs)
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 289, in _configure
    _initialize_components(kwargs.get("auto_instrumentation_version"))
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 233, in _initialize_components
    trace_exporters, metric_exporters, log_exporters = _import_exporters(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 201, in _import_exporters
    for (exporter_name, exporter_impl,) in _import_config_components(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 170, in _import_config_components
    raise RuntimeError(
RuntimeError: Requested component 'otlp_proto_grpc' not found in entry points for 'opentelemetry_metrics_exporter'
Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 121, in initialize
    _load_configurators()
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 109, in _load_configurators
    raise exc
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 105, in _load_configurators
    entry_point.load()().configure(auto_instrumentation_version=__version__)  # type: ignore
 File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 273, in configure
    self._configure(**kwargs)
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 289, in _configure
    _initialize_components(kwargs.get("auto_instrumentation_version"))
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 233, in _initialize_components
    trace_exporters, metric_exporters, log_exporters = _import_exporters(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 201, in _import_exporters
    for (exporter_name, exporter_impl,) in _import_config_components(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 170, in _import_config_components
    raise RuntimeError(
RuntimeError: Requested component 'otlp_proto_grpc' not found in entry points for 'opentelemetry_metrics_exporter'

Additional context I've tried adding the missing otlp_proto_grpc and rebuilt the init container docker image, and tried using that image. But it then throws this error:

Configuration of configurator failed
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 105, in _load_configurators
    entry_point.load()().configure(auto_instrumentation_version=__version__)  # type: ignore
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 344, in configure
    self._configure(**kwargs)
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 360, in _configure
    _initialize_components(kwargs.get("auto_instrumentation_version"))
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 304, in _initialize_components
    trace_exporters, metric_exporters, log_exporters = _import_exporters(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 272, in _import_exporters
    for (exporter_name, exporter_impl,) in _import_config_components(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 245, in _import_config_components
    component_impl = entry_point.load()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2471, in load
    return self.resolve()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2477, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/otel-auto-instrumentation/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py", line 18, in <module>
    from grpc import ChannelCredentials, Compression
  File "/otel-auto-instrumentation/grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/otel-auto-instrumentation/grpc/_compression.py", line 15, in <module>
    from grpc._cython import cygrpc
ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/otel-auto-instrumentation/grpc/_cython/__init__.py)
Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 121, in initialize
    _load_configurators()
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 109, in _load_configurators
    raise exc
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 105, in _load_configurators
    entry_point.load()().configure(auto_instrumentation_version=__version__)  # type: ignore
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 344, in configure
    self._configure(**kwargs)
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 360, in _configure
    _initialize_components(kwargs.get("auto_instrumentation_version"))
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 304, in _initialize_components
    trace_exporters, metric_exporters, log_exporters = _import_exporters(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 272, in _import_exporters
    for (exporter_name, exporter_impl,) in _import_config_components(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 245, in _import_config_components
    component_impl = entry_point.load()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2471, in load
    return self.resolve()
  File "/otel-auto-instrumentation/pkg_resources/__init__.py", line 2477, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/otel-auto-instrumentation/opentelemetry/exporter/otlp/proto/grpc/metric_exporter/__init__.py", line 18, in <module>
    from grpc import ChannelCredentials, Compression
  File "/otel-auto-instrumentation/grpc/__init__.py", line 22, in <module>
    from grpc import _compression
  File "/otel-auto-instrumentation/grpc/_compression.py", line 15, in <module>
    from grpc._cython import cygrpc
ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/otel-auto-instrumentation/grpc/_cython/__init__.py)

I've also tried updating all pip packages and dependencies to utilise opentelemetry-exporter-otlp-proto version 1.13.0 & opentelemetry-distro version 0.34b0 in this file: https://github.com/open-telemetry/opentelemetry-operator/blob/main/autoinstrumentation/python/requirements.txt
But I get the same error Requested component 'otlp_proto_grpc' not found.

To rule out other issue, I've tested dotnet auto-injeciton, and that works fine with no issues.

Regards

dominthomas commented 2 years ago

It looks like this issue was fixed as part of this PR: #2748 But that doesn't seem to be the case 🤔 @srikanthccv - any ideas?

srikanthccv commented 2 years ago

I don't think the operator supports the grpc for the reasons stated in the requirements.txt file. And there is no metrics HTTP library at the moment but that doesn't raise an issue.

dominthomas commented 2 years ago

Yes, you're right, I did see that comment - so not sure what's causing this. Is it a case of dsiabling opentelemetry_metrics_exporter? Not sure if that's possible or if it's a req for that.

srikanthccv commented 2 years ago

If you are using the operator you would have to use the HTTP exporter for traces and logs and set the exporter env to otlp_proto_http instead of otlp_proto_grpc

dominthomas commented 2 years ago

Tried that, same error... looks like it's expecting otlp_proto_grpc for metrics regardless.

aabmass commented 2 years ago

Tried that, same error... looks like it's expecting otlp_proto_grpc for metrics regardless.

I was able to get it working with @srikanthccv's workaround, in the instrumentation manifest:

  python:
    env:
    - name: OTEL_METRICS_EXPORTER
      value: none

One issue is the opentelemetry-operator doesn't have a build of 0.34b0 and it is by default using very old 0.32b0. We need to bump the deps and have them build some new images, opened https://github.com/open-telemetry/opentelemetry-operator/pull/1147

The other problem is the operator should probably set OTEL_METRICS_EXPORTER=none until https://github.com/open-telemetry/opentelemetry-python/issues/2447 is fixed to prevent it from crashing. I don't the user should need to do this manual workaround. Opened https://github.com/open-telemetry/opentelemetry-operator/pull/1149

vasireddy99 commented 2 years ago

+1 I was able to get working by setting OTEL_METRICS_EXPORTER:NONE from the last couple of releases.

aabmass commented 2 years ago

This should be fixed in the latest release https://github.com/open-telemetry/opentelemetry-operator/releases/tag/v0.61.0. However you'll still have to change the OTLP endpoint for traces to use port 4318 as described in https://github.com/open-telemetry/opentelemetry-operator/issues/924, but that is a separate issue.