500 internal error from the :9913/metrics endpoint

[remimikalsen]

I've got a few vts-exporters running and working perfectly. But suddenly my most important exporter started failing.

The vts-exporter in question export stats from an nginx server which distributes traffic to several upstream servers on different domains. When I added yet another "vhost" (an upstream Passbolt server), my vts-exporter started failing.

When I access the metrics endpoint with wget, I simply get 500 internal error (for this test, I exposed port 9913 on my docker host which forwards requests to port 9913 on my vts-exporter):

wget localhost:9913/metrics
--2019-12-15 20:11:27--  http://localhost:9913/metrics
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:9913... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2019-12-15 20:11:27 ERROR 500: Internal Server Error.

Note that another vts exporter works fine with this same test.

I can't find any logs on the server, only the docker logs. Nothing in those logs indicates internal server errors.

Log extract:

2019/12/15 20:22:38 Starting nginx_vts_exporter (version=, branch=, revision=)
2019/12/15 20:22:38 Build context (go=go1.12.5, user=, date=)
2019/12/15 20:22:38 Starting Server at : :9913
2019/12/15 20:22:38 Metrics endpoint: /metrics
2019/12/15 20:22:38 Metrics namespace: nginx
2019/12/15 20:22:38 Scraping information from : http://nginx-vts-status/status/format/json

The scraping endpoint on the Nginx-server in question returns valid JSON for the vts exporter to consume. The file size is 43028 bytes.

Can the file size be a problem? What would be the recommended way of debugging this further?

Edit Testing further I've come to the conclusion that the Passbolt image is the problem. I've tried disabling containers in turn, and always when enabling Passbolt, things stop working. This is my Passbolt config:

  passblt:
    container_name: passblt
    image: passbolt/passbolt:2.12.0-debian
    tty: true
    depends_on:
      - passbolt_db
    networks:
      - frontend
      - backend
      - email
    volumes:
      - passbolt_gpg:/var/www/passbolt/config/gpg
      - passbolt_images:/var/www/passbolt/webroot/img/public
    tmpfs:
      - /run
    command: ["/usr/bin/wait-for.sh", "-t", "0", "passbolt_db:3306", "--", "/docker-entrypoint.sh"]
    environment:
      - TZ=${DOCKER_TZ}
      - APP_FULL_BASE_URL=${PASSBOLT_APP_FULL_BASE_URL}
      - DATASOURCES_DEFAULT_HOST=${PASSBOLT_DATASOURCES_DEFAULT_HOST}
      - DATASOURCES_DEFAULT_PORT=${PASSBOLT_DATASOURCES_DEFAULT_PORT}
      - DATASOURCES_DEFAULT_USERNAME=${PASSBOLT_MYSQL_USER}
      - DATASOURCES_DEFAULT_PASSWORD=${PASSBOLT_MYSQL_PASSWORD}
      - DATASOURCES_DEFAULT_DATABASE=${PASSBOLT_MYSQL_DATABASE}
      - PASSBOLT_REGISTRATION_PUBLIC=${PASSBOLT_REGISTRATION_PUBLIC}
      - EMAIL_TRANSPORT_DEFAULT_HOST=smtp-relay
      - EMAIL_TRANSPORT_DEFAULT_PORT=25
      - EMAIL_DEFAULT_FROM=${NOTIFICATION_EMAIL}
      - VIRTUAL_HOST=${PASSBOLT_DOMAIN}
      - VIRTUAL_PORT=80
      - VIRTUAL_PROTO=http
      - LETSENCRYPT_HOST=${PASSBOLT_DOMAIN}
      - LETSENCRYPT_EMAIL=${NOTIFICATION_EMAIL}
    labels:
      co.elastic.logs/disable: false

I've tried disabling tty: true - to no avail. I'm not sure why tty is needed, but I got it from the official Passbolt docker-compose documentation.

One thing I do notice in my "http://nginx-vts-status/status/format/json" , is that the Passbolt Nginx UpstreamZone, as the only one, has two entries for "server: 127.0.0.1:80". Maybe this causes the exporter to crash?

Edit2 I've confirmed that duplicate "127.0.0.1:80 down" entries in my Passbolt UpstreamZone made the vts-exporter metric endpoint crash. I solved the problem by removing the duplicate entry from my nginx-config. The cause of my duplicate entries was the default nginx-gen template. Editing the template solved he issue.

Nevertheless, it would be nice if the vts-exporter handled duplicate entries, epecially when the entries are reported as "down".