🐛 Bug Report: No module named 'opentelemetry.semconv.ai'

[reiyw]

Which component is this bug for?

LLM Semantic Conventions

📜 Description

Due to an inconsistency in the packaging methods between opentelemetry-semantic-conventions and opentelemetry-semantic-conventions-ai, importing opentelemetry.semconv.ai may fail in certain environments.

👟 Reproduction steps

Reproduction repository: https://github.com/reiyw/opentelemetry-semconv-ai-repro

Run the following code:

from traceloop import sdk

👍 Expected behavior

It should be imported without any issues.

👎 Actual Behavior with Screenshots

The following error is output:

Traceback (most recent call last):
  File "/home/ryo.takahashi/.cache/bazel/_bazel_ryo.takahashi/73da590b5d6f7b4220b10a2aad9d98d9/execroot/_main/bazel-out/k8-fastbuild/bin/main.runfiles/_main/main.py", line 11, in <module>
    from traceloop import sdk
  File "/home/ryo.takahashi/.cache/bazel/_bazel_ryo.takahashi/73da590b5d6f7b4220b10a2aad9d98d9/execroot/_main/bazel-out/k8-fastbuild/bin/main.runfiles/rules_python~~pip~my_deps_311_traceloop_sdk/site-packages/traceloop/sdk/__init__.py", line 14, in <module>
    from traceloop.sdk.metrics.metrics import MetricsWrapper
  File "/home/ryo.takahashi/.cache/bazel/_bazel_ryo.takahashi/73da590b5d6f7b4220b10a2aad9d98d9/execroot/_main/bazel-out/k8-fastbuild/bin/main.runfiles/rules_python~~pip~my_deps_311_traceloop_sdk/site-packages/traceloop/sdk/metrics/metrics.py", line 10, in <module>
    from opentelemetry.semconv.ai import Meters
ModuleNotFoundError: No module named 'opentelemetry.semconv.ai'

🤖 Python Version

3.11

📃 Provide any additional context for the Bug.

The root cause of this issue is the inconsistency in the packaging methods between opentelemetry-semantic-conventions and opentelemetry-semantic-conventions-ai. In opentelemetry-semantic-conventions, an empty opentelemetry/semconv/__init__.py is placed, which causes the opentelemetry.semconv package to be recognized as a regular package. On the other hand, in opentelemetry-semantic-conventions-ai, the absence of opentelemetry/semconv/__init__.py leads to the opentelemetry.semconv package being recognized as a namespace package.

It is not a problem in the style of consolidating packages in one place (the usual style using pip or poetry) for a package to be a regular package and a namespace package at the same time, but in general it can cause import problems. The following is the directory structure of the opentelemetry.semconv package when traceloop-sdk is installed using pip:

- site-packages/
    - opentelemetry/
        - semconv/
            - __init__.py  # <- empty, from opentelemetry-semantic-conventions
            - ai/
                - __init__.py  # <- from opentelemetry-semantic-conventions-ai

In this situation, the opentelemetry.semconv.ai package is recognized as a subpackage of the opentelemetry.semconv package, so it can be imported without any problems.

On the other hand, consider the following directory structure:

- site-packages-1/
    - opentelemetry/
        - semconv/
            - __init__.py  # <- empty, from opentelemetry-semantic-conventions
- site-packages-2/
    - opentelemetry/
        - semconv/
            - ai/
                - __init__.py  # <- from opentelemetry-semantic-conventions-ai

In this situation, the package logic is broken. In the site-packages-1 directory, opentelemetry.semconv is a regular package that does not include the ai namespace, so if it is loaded first, the import of opentelemetry.semconv.ai will fail. This directory structure may seem strange, but it is the structure adopted by rules_python (the Python toolchain for Bazel), and namespace packages are a mechanism that allows for this kind of structure.

I think the fundamental solution to this problem is one of the following two:

• Request the maintainer of opentelemetry-semantic-conventions to make the opentelemetry.semconv package a namespace package.
• Restructure the opentelemetry.semconv.ai package. For example, distributing it as the opentelemetry.semconv_ai package. opentelemetry is an officially namespace package, so it should work correctly.

There are workarounds for this at runtime, as shown in my reproduction repository, but it would be better to resolve the underlying problem.

👀 Have you spent some time to check if this bug has been raised before?

• [X] I checked and didn't find similar issue

Are you willing to submit PR?

None

[nirga]

Thanks for reporting @reiyw! Can you confirm that adding an empty __init__.py to opentelemetry/semconv under the opentelemetry-semconv-conventions-ai solves this?

[reiyw]

adding an empty __init__.py to opentelemetry/semconv under the opentelemetry-semconv-conventions-ai

That won't solve the problem. Since the opentelemetry.semconv package is a regular package in opentelemetry-semantic-conventions, if it is loaded first, this issue will occur regardless of whether the opentelemetry.semconv package is a regular package or a namespace package in opentelemetry-semantic-conventions-ai.

[nirga]

@reiyw how do you think we should resoive it? rename the package to be under opentelemetry/semconv-ai or something?

[reiyw]

@nirga That is one solution. Another solution is to have the opentelemetry.semconv package distributed by opentelemetry-semantic-conventions changed from a regular package to a namespace package.

Either way it would be a breaking change, but the way of restructuring the opentelemetry.semconv.ai package would be easier for you to control the situation. As an example, the steps for restructuring would be as follows:

1. Move the content of packages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv/ai to packages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai.
2. Add the path opentelemetry/semconv_ai to the tool.poetry.packages.include.
3. Put code in semconv/ai/__init__.py to re-export the symbols in semconv_ai and issue a DeprecationWarning. At this point, the user can import the same package with import opentelemetry.semconv.ai and import opentelemetry.semconv_ai, but will be prompted to switch to the latter style.
4. Remove opentelemetry/semconv/ai after a certain migration period.