search

percona / pmm

Percona Monitoring and Management: an open source database monitoring, observability and management tool
https://www.percona.com/software/database-tools/percona-monitoring-and-management
GNU Affero General Public License v3.0
679 stars 131 forks source link

2.33.0 Docker image error (both after upgrade, and clean installation) #1630

Open ernstae opened 1 year ago

ernstae commented 1 year ago

Description

I have yet to run the container image for 2.33.0 successfully. I've tested upgrading (using my data container + changing the server image which has worked flawlessly in the past). I've also tested a new installation and at this point, haven't been able to successfully start the pmm-server containers. The issue appears to occur at this point of the instantiation: pmm-server | 2023-01-26 06:21:04,252 INFO exited: qan-api2 (exit status 1; not expected) and grafana goes into a crashloop cycle after reporting a

Expected Results

Service starts correctly and presents login screen

Actual Results

500 Error in Nginx (bad gateway).

Resulting in unhealthy containers

# docker ps -a
CONTAINER ID   IMAGE                                          COMMAND                  CREATED         STATUS                     PORTS                                      NAMES
1905e3414872   docker.redacted.net/percona/pmm-server:2.33.0   "/bin/bash -c 'ln -s…"   6 minutes ago   Up 6 minutes (unhealthy)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   pmm-server
3145d1d3f79e   docker.redacted.net/percona/pmm-server:2.33.0   "/bin/bash -c 'chown…"   7 minutes ago   Exited (0) 6 minutes ago                                              pmm-data

All pointing to grafana not loading the pmm-app plugin

Version

PMM Server 2.33.0

Steps to reproduce

Using this docker-compose.yaml file:

version: '2'

services:
  pmm-data:
    image: docker.redacted.net/percona/pmm-server:2.33.0
    container_name: pmm-data
    volumes:
      - /u01/srv:/srv

    command:
      - /bin/bash
      - -c
      - |
        chown -R root:clickhouse /srv/clickhouse
        chown -R grafana:grafana /srv/grafana
        chown -R pmm:pmm /srv/logs /srv/prometheus /srv/alertmanager
        chown -R postgres /srv/postgres
        chown -R postgres /srv/logs/postgresql.log
        /bin/true

  pmm-server:
    image: docker.redacted.net/percona/pmm-server:2.33.0
    container_name: pmm-server
    ports:
      - '80:80'
      - '443:443'
    restart: always
    environment:
      - DISABLE_UPDATES=true
      - DISABLE_TELEMETRY=true
      - DATA_RETENTION=2160h
    command:
      - /bin/bash
      - -c
      - |
        ln -s /srv/grafana /usr/share/grafana/data; grafana-cli --homepath /usr/share/grafana admin reset-admin-password 'redactedpassword'
        /opt/entrypoint.sh
    volumes_from:
      - pmm-data

Relevant logs

# from grafana.log
logger=sqlstore t=2023-01-26T06:43:09.279464905Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=migrator t=2023-01-26T06:43:09.294504382Z level=info msg="Starting DB migrations"
logger=migrator t=2023-01-26T06:43:09.299234637Z level=info msg="migrations completed" performed=0 skipped=452 duration=546.206µs
logger=plugin.finder t=2023-01-26T06:43:09.350125134Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
logger=plugin.loader t=2023-01-26T06:43:09.599976665Z level=warn msg="Skipping loading plugin due to problem with signature" pluginID=pmm-pt-summary-datasource status=unsigned
logger=plugin.loader t=2023-01-26T06:43:09.600021765Z level=warn msg="Skipping loading plugin due to problem with signature" pluginID=pmm-update status=unsigned
logger=plugin.loader t=2023-01-26T06:43:09.600045666Z level=warn msg="Skipping loading plugin due to problem with signature" pluginID=pmm-pt-summary-panel status=unsigned
logger=plugin.loader t=2023-01-26T06:43:09.600055366Z level=warn msg="Skipping loading plugin due to problem with signature" pluginID=pmm-qan-app-panel status=unsigned
logger=plugin.loader t=2023-01-26T06:43:09.600064166Z level=warn msg="Skipping loading plugin due to problem with signature" pluginID=pmm-check-panel-home status=unsigned
logger=plugin.loader t=2023-01-26T06:43:09.600071666Z level=warn msg="Skipping loading plugin due to problem with signature" pluginID=vertamedia-clickhouse-datasource status=unsigned
logger=plugin.loader t=2023-01-26T06:43:09.600096866Z level=warn msg="Skipping loading plugin due to problem with signature" pluginID=grafana-polystat-panel status=unsigned
logger=plugin.loader t=2023-01-26T06:43:09.600114366Z level=warn msg="Skipping loading plugin due to problem with signature" pluginID=pmm-app status=unsigned
logger=plugin.loader t=2023-01-26T06:43:09.600164667Z level=info msg="Plugin registered" pluginID=petrslavotinek-carpetplot-panel
logger=plugin.loader t=2023-01-26T06:43:09.600181567Z level=info msg="Plugin registered" pluginID=grafana-worldmap-panel
logger=plugin.loader t=2023-01-26T06:43:09.600189067Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel
logger=plugin.loader t=2023-01-26T06:43:09.600195967Z level=info msg="Plugin registered" pluginID=yesoreyeram-boomtable-panel
logger=plugin.loader t=2023-01-26T06:43:09.600201868Z level=info msg="Plugin registered" pluginID=jdbranham-diagram-panel
logger=plugin.loader t=2023-01-26T06:43:09.600207368Z level=info msg="Plugin registered" pluginID=camptocamp-prometheus-alertmanager-datasource
logger=plugin.loader t=2023-01-26T06:43:09.600214868Z level=info msg="Plugin registered" pluginID=natel-discrete-panel
logger=secrets t=2023-01-26T06:43:09.600474271Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=query_data t=2023-01-26T06:43:09.603518606Z level=info msg="Query Service initialization"
logger=live.push_http t=2023-01-26T06:43:09.609171073Z level=info msg="Live Push Gateway initialization"
logger=infra.usagestats.collector t=2023-01-26T06:43:09.706840418Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=provisioning t=2023-01-26T06:43:09.825091305Z level=error msg="Failed to provision plugins" error="app provisioning error: plugin not installed: \"pmm-app\""
Failed to start grafana. error: app provisioning error: plugin not installed: "pmm-app"
app provisioning error: plugin not installed: "pmm-app"

Code of Conduct

ademidoff commented 1 year ago

I wouldn't call it a bug for the simple reason that the way the data container is created does not follow what our docs recommend.

The error app provisioning error: plugin not installed: "pmm-app" means that PMM could not find the contents of the plugin and therefore failed to provision it.

One thing I highly recommend is to provision the data volume following our instructions and possibly remove the ln -s command from the pmm-server container - it should solve it.

denisok commented 1 year ago

another lifehack you could use is to pass GF password/user for admin as an env var GF_SECURITY_ADMIN_USER: https://github.com/percona/percona-helm-charts/blob/main/charts/pmm/templates/statefulset.yaml#L78

Or as a file (although I didn't verified that): https://grafana.com/docs/grafana/latest/setup-grafana/configure-docker/#configure-grafana-with-docker-secrets

GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/admin_password