Having issues installing the plugin in a k8s env

[Manan-Kothari]

Hi, we are using a custom forked version of kong where we build our own kong image and when installing the ddtrace plugin in the dockerfile and then enabling it, getting this error

2024/01/24 18:09:08 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:621: error loading plugin schemas: on plugin 'ddtrace': ddtrace plugin is enabled but not installed;
no plugin found
stack traceback:
    [C]: in function 'assert'
    /usr/local/share/lua/5.1/kong/init.lua:621: in function 'init'
    init_by_lua:3: in main chunk
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:621: error loading plugin schemas: on plugin 'ddtrace': ddtrace plugin is enabled but not installed;
no plugin found
stack traceback:
    [C]: in function 'assert'
    /usr/local/share/lua/5.1/kong/init.lua:621: in function 'init'
    init_by_lua:3: in main chunk

Our dockerfile looks like this

FROM kong:3.3.0-ubuntu
USER root

ARG POETRY_HTTP_BASIC_P44_USERNAME
ARG POETRY_HTTP_BASIC_P44_PASSWORD

ENV POETRY_HTTP_BASIC_P44_USERNAME=${POETRY_HTTP_BASIC_P44_USERNAME} \
    POETRY_HTTP_BASIC_P44_PASSWORD=${POETRY_HTTP_BASIC_P44_PASSWORD} \
    PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    PIP_DEFAULT_TIMEOUT=100 \
    POETRY_NO_INTERACTION=1 \
    POETRY_VIRTUALENVS_IN_PROJECT=true

# Make sure home directory exists
RUN mkdir -p /home/kong

# Install system deps
RUN apt-get update \
&& apt-get -y install python3-pip musl-dev libffi-dev gcc g++ file make librdkafka1 librdkafka-dev \
&& apt-get clean \
&& pip3 install --upgrade pip setuptools poetry==1.4.1

# Install Python dependencies
COPY ./pyproject.toml ./poetry.lock /home/kong/
RUN cd /home/kong  \
&& poetry export --without dev --with-credentials --no-ansi --without-hashes --format=requirements.txt > requirements.txt  \
&& pip3 install -r requirements.txt  \
&& rm pyproject.toml poetry.lock requirements.txt
# Install dd-trace-plugin
RUN luarocks install kong-plugin-ddtrace
# Copy Kong plugins
COPY ./custom_plugins/py /home/kong/plugins
COPY ./bin /home/kong/bin
RUN chmod -R 775 /home/kong/plugins /home/kong/bin

RUN chown -R kong:kong /home/kong

# Temporary patch for https://github.com/Kong/kong-python-pdk/issues/120
COPY ./patches/kong_pdk/server.patch /usr/local/lib/python3.10/dist-packages/kong_pdk/
RUN patch /usr/local/lib/python3.10/dist-packages/kong_pdk/server.py < /usr/local/lib/python3.10/dist-packages/kong_pdk/server.patch

# Temporary patch for https://github.com/Kong/kong/issues/11173
COPY ./patches/kong/runloop/handler.patch /usr/local/share/lua/5.1/kong/runloop/
RUN patch /usr/local/share/lua/5.1/kong/runloop/handler.lua < /usr/local/share/lua/5.1/kong/runloop/handler.patch

# Initialize Kong
USER kong
ENTRYPOINT ["/home/kong/bin/docker-entrypoint.sh"]
EXPOSE 8000 8443 8001 8444
STOPSIGNAL SIGQUIT
HEALTHCHECK --interval=10s --timeout=10s --retries=10 CMD kong health
CMD ["kong", "docker-start"]

and then in our helm chart we have the following

env:
    # All custom plugin names must be present here
    PLUGINS: "bundled,request_to_shared_context,auth_session_to_headers,cookie_token_to_header,sleuth_trace_header_injector,statsd,cors_filter,rate_limit,rate_limit_for_tenant,xss_filter,api_logger,disable_auth,request_terminator,request_sanitizer,propagate_query_parameters,access_group_allowlist_filter,metrics_reporter,ddtrace"
    LOG_LEVEL: "debug"
  dblessConfig:
    config:
      _format_version: "2.23.0"
      plugins:
        - name: ddtrace
          config:
            service_name: p44-kong
            agent_host: "{vault://env/dd-agent-host}"
            environment: 'qa-integration'

we are setting the env variables as well for the host, etc but it looks it can't even find the plugin

When we run this locally and check if the plugin is available in the kong config, we do see it (using that same dockerfile)

This is our config locally and can see the plugin

p44-kong-config.json

[dmehala]

Hi @Manan-Kothari,

To ensure there were no issues with luarock, I constructed a minimal Dockerfile based on the same Kong image. Successfully, I loaded, ran, and generated traces as expected.

For the record, here is the dockerfile:

FROM kong:3.3.0-ubuntu
user root

ENV KONG_DATABASE=off
ENV KONG_DECLARATIVE_CONFIG_STRING='{"_format_version":"1.1", "services":[{"host":"httpbin.org","port":443,"protocol":"https", "routes":[{"paths":["/"]}]}],"plugins":[{"name":"ddtrace", "config":{"service_name":"dmehala-kong"}}]}'
ENV KONG_PLUGINS=bundled,ddtrace

# Install dd-trace-plugin
RUN luarocks install kong-plugin-ddtrace

For now, I don't suspect a luarock issue. It appears that enabling the ddtrace plugin in your custom forked version of Kong an error is encountered:

Hi, we are using a custom forked version of kong where we build our own kong image and when installing the ddtrace plugin in the dockerfile and then enabling it, getting this error

2024/01/24 18:09:08 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:621: error loading plugin schemas: on plugin 'ddtrace': ddtrace plugin is enabled but not installed; no plugin found stack traceback: [C]: in function 'assert' /usr/local/share/lua/5.1/kong/init.lua:621: in function 'init' init_by_lua:3: in main chunk

The plugin do not appear in p44-kong-config.json. However, there is a datadog plugin which is probably the plugin developed by Kong for Datadog metric. Our tracing module for Kong is named kong-plugin-ddtrace and its latest version is 0.1.2

It seems Kong is unable to load kong-plugin-ddtrace plugin. I observed that the Helm chart sets the PLUGINS environment variable, while Kong fetches the list of dependencies to load from KONG_PLUGINS. For now, let's try KONG_PLUGINS instead.

Please, also review Kong's documentation on deploying custom plugins.

My team and I will continue to investigate the issue. Nevertheless, as a precautionary measure, I highly recommend opening a Datadog support ticket to ensure comprehensive assistance and a swift resolution.

[ebatesdev]

@dmehala hello, I am also working on this team and with this issue. I think we provided the wrong config file initially. Here is the correct one. In this config you can see ddtrace show up in ln 591 "ddtrace": { "priority": 100000, "version": "0.1.2" }, and ln 273 "loaded_plugins": { "cookie_token_to_header": true, "request_sanitizer": true, "propagate_query_parameters": true, "access_group_allowlist_filter": true, "metrics_reporter": true, "ddtrace": true, p44-kong-config2.json

also regarding the env variables, I was under the impression that Any variable under env will be prefixed with KONG_ (see https://github.com/Kong/charts/blob/main/charts/kong/templates/_helpers.tpl#L1172). DD_AGENT_HOST will be set as KONG_DD_AGENT_HOST.

is this not the case?

[dmehala]

Handled through Zendesk.