search

immauss / openvas

Containers for running the Greenbone Vulnerability Manager. Run as a single container with all services or separate single applications containers via docker-compose.
GNU Affero General Public License v3.0
337 stars 97 forks source link

[BUG] Healthchecks report no issue but container remains unhealthy #234

Closed mnaismith closed 7 months ago

mnaismith commented 7 months ago

I've been running your excellent Openvas container for some time. This week I have the below issue.

I am running Ubuntu 22.04.3 with all system updates applied.

I pull down latest immauss/openvas image.

I run with command

docker run --detach --publish 127.0.0.1:8080:9392 --publish 127.0.0.1:9390:9390 -e SKIPSYNC=true -e PASSWORD="blabla" --volume openvas:/data --name openvas immauss/openvas

I watch the status of the container and it never becomes healthy.

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c0691e402070 immauss/openvas "/scripts/start.sh" 23 minutes ago Up 23 minutes (unhealthy) 127.0.0.1:9390->9390/tcp, 127.0.0.1:8080->9392/tcp openvas

Logs attached.

logfile.txt

immauss commented 7 months ago

That is strange. The logs look perfectly normal and the output from the health check script shows no issues. I'll do some testing here and see if I can replicate.

And thanks for the subscription! Much appreciated.

-Scott

immauss commented 7 months ago

I've done several rounds of testing with this, and I'm not able to reproduce it. Is the running container useable despite the unhealthy indication?

Have you tried restarting docker?

The really odd part is that the logs show the script is completing successfully. If that happens, then it should show as healthy on a "docker ps" (after 5 minutes). I've been staring at the healthcheck.sh trying to think of some scenario that would make it exit with something other than 0 and still give the "all good" message, but I'm coming up empty.

Can you send me the out from the following:

docker exec -it openvas bash -x "/scripts/healthcheck.sh"

If the healthcheck.sh is doing something funky, this should help me find it.

Thanks, Scott

grandaor commented 7 months ago

I've the same thing:

$ docker exec -it openvas bash -x "/scripts/healthcheck.sh"
++ cat /usr/local/etc/running-as
+ FUNC=single
++ df -h /
++ tr -d %
++ awk '/overlay/{ if ( $5 > 95 ) print $4}'
+ HIGHROOT=
++ df /
++ tr -d %
++ awk '/overlay/{print $4}'
+ ROOTSPC=32287300
+ '[' -z ']'
++ df -h
++ tr -d %
++ awk '/data/{ if ( $5 > 95 ) print $4}'
+ HIGHDATA=
++ df
++ tr -d %
++ awk '/data/{print $4}'
+ DATASPC=32287300
+ '[' -z ']'
+ case $FUNC in
+ FAIL=0
++ su -c 'gvmd --get-scanners' gvm
++ awk '/OpenVAS/{print  $1}'
+ UUID=08b69003-5fc2-4037-a479-93b440211c73
+ su -c 'gvmd --verify-scanner=08b69003-5fc2-4037-a479-93b440211c73' gvm
+ grep OpenVAS
Scanner version: OpenVAS 22.7.7.
+ '[' 0 -eq 1 ']'
+ nmap -p 9390 localhost
+ grep -qs '9390.*open'
+ '[' 0 -eq 2 ']'
+ curl -f http://localhost:9392/
<!doctype html><html><head><link rel="icon" href="/img/favicon.png" type="image/png"/><title>Greenbone Security Assistant</title><meta charset="UTF-8"><meta name="viewport" content="width=device-width,initial-scale=1"><script type="text/javascript" src="/config.js"></script><script defer="defer" src="/static/js/main.5e6dfc4c.js"></script><link href="/static/css/main.d6631c26.css" rel="stylesheet"></head><body><div id="app"></div><div id="portals"></div></body></html>+ '[' 0 -eq 3 ']'
+ redis-cli -s /run/redis/redis.sock ping
PONG
+ '[' 0 -eq 4 ']'
+ nmap -p 5432 localhost
+ grep -qs '5432.*open'
+ '[' 0 -eq 5 ']'
+ '[' 0 -ne 0 ']'
+ echo ' Healthchecks completed with no issues.'

I've just pull the latest image and still unhealthy.

I've test with a clean volume and still unhealthy.

$ docker ps
CONTAINER ID   IMAGE                                     COMMAND                  CREATED          STATUS                      PORTS                                                          NAMES
036c99b9142b   immauss/openvas                           "/scripts/start.sh"      15 minutes ago   Up 15 minutes (unhealthy)                                                                  openvas
immauss commented 7 months ago

@grandaor Are you able to login and use the container despite the "unhealthy" status?

Thanks, Scott

grandaor commented 7 months ago

yes without my traefik configuration

but with my traefik that doesn't work because de dock is still unhealthy :(

immauss commented 7 months ago

@mnaismith Are you also using traefik ?

immauss commented 7 months ago

@grandaor Can you walk me through how you have traefik setup? I've never really used it. If you can help me replicate your setup, maybe I can find the issue.

Thanks, Scott

grandaor commented 7 months ago

I don't think this issue is related to traefik because without it, the dock openvas is still unhealthy...

my traefik setup :

version: '3.3'
services:
  traefik:
    image: traefik:latest
    env_file:
    - .env
    container_name: traefik
    restart: unless-stopped
    ports:
      - target: 80
        published: 80
        mode: host
      - target: 443
        published: 443
        mode: host
    labels:
      - traefik.enable=true
      - traefik.docker.network=traefik-public
      - traefik.constraint-label=traefik-public
      - traefik.http.middlewares.admin-auth.basicauth.users=${USERNAME?Variable not set}:${HASHED_PASSWORD?Variable not set}
      - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
      - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true
      - traefik.http.routers.traefik-public-http.rule=Host(`${DOMAIN?Variable not set}`)
      - traefik.http.routers.traefik-public-http.entrypoints=http
      - traefik.http.routers.traefik-public-http.middlewares=https-redirect
      - traefik.http.routers.traefik-public-https.rule=Host(`${DOMAIN?Variable not set}`)
      - traefik.http.routers.traefik-public-https.entrypoints=https
      - traefik.http.routers.traefik-public-https.tls=true
      - traefik.http.routers.traefik-public-https.service=api@internal
      - traefik.http.routers.traefik-public-https.tls.certresolver=le
      - traefik.http.routers.traefik-public-https.middlewares=admin-auth
      - traefik.http.services.traefik-public.loadbalancer.server.port=8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./certificates:/certificates
      - /var/log/traefik/:/logs/
    command:
      - --providers.docker
      - --providers.docker.constraints=Label(`traefik.constraint-label`, `traefik-public`)
      - --providers.docker.exposedbydefault=false
      - --entrypoints.http.address=:80
      - --entrypoints.https.address=:443
      - --certificatesresolvers.le.acme.email=${EMAIL?Variable not set}
      - --certificatesresolvers.le.acme.storage=/certificates/acme.json
      - --certificatesresolvers.le.acme.tlschallenge=true
      - --accesslog
      - --log
      - --accesslog.filePath=/logs/access.log
      - --api
    networks:
      - traefik-public

networks:
  traefik-public:
    external: true

my openvas setup :

version: '3.3'

services:

  openvas:
    image: immauss/openvas
    container_name: openvas
    env_file:
      - .env
    restart: unless-stopped
    environment:
      PASSWORD: ${PASSWORD?Variable not set}
      HTTPS: false
      PUBLIC_HOSTNAME: ${URL_T?Variable not set}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /etc/localtime:/etc/localtime:ro
      - ./data:/data
    networks:
      - traefik-public
    labels:
      - traefik.enable=true
      - traefik.docker.network=traefik-public
      - traefik.constraint-label=traefik-public
      - traefik.http.routers.openvas-http.rule=Host("${URL_T?Variable not set}")
      - traefik.http.routers.openvas-http.entrypoints=http
      - traefik.http.routers.openvas-http.middlewares=https-redirect
      - traefik.http.routers.openvas-https.rule=Host("${URL_T?Variable not set}")
      - traefik.http.routers.openvas-https.entrypoints=https
      - traefik.http.routers.openvas-https.tls=true
      - traefik.http.routers.openvas-https.tls.certresolver=le
      - traefik.http.middlewares.hsts-headers.headers.stsSeconds=315360000
      - traefik.http.middlewares.hsts-headers.headers.stsPreload=true
      - traefik.http.middlewares.hsts-headers.headers.stsIncludeSubdomains=true
      - traefik.http.middlewares.hsts-headers.headers.customFrameOptionsValue=SAMEORIGIN
      - traefik.http.middlewares.hsts-headers.headers.sslRedirect=true
      - traefik.http.middlewares.hsts-headers.headers.forceSTSHeader=true
      - traefik.http.middlewares.hsts-headers.headers.referrerPolicy=same-origin
      - traefik.http.routers.openvas-https.middlewares=hsts-headers
      - traefik.http.services.openvas.loadbalancer.server.port=9392
networks:
  traefik-public:
    external: true
immauss commented 7 months ago

OK ... not sure what I did differently today from yesterday, but I'm seeing the same issue now on a straight docker. Let me see what I can find now.

Thanks, -Scott

grandaor commented 7 months ago

Thanks for your help @immauss :)

immauss commented 7 months ago

FOUND IT!

It's a timing thing. Which explains why I see it sometimes and not others.

The timeout in the Dockerfile for the healthcheck is set to 10s.

Today, my VM is taking 11.2 seconds to complete the healthcheck. Yesterday it was probably less than 10. I'm going to up the timeout to 30s, which should be long enough for most environments.

It will take about an hour to rebuild and push to docker. The version will be 22.4.35.

-Scott

grandaor commented 7 months ago

oh yeah !!!

grandaor commented 7 months ago

here, it's 13.593s to complete the healthcheck

immauss commented 7 months ago

Build Kit Cache for the WIN! 22.4.35 & latest are now both updated with the longer timeout.

Please let me know if this resolves the issue for you.

Thanks, Scott

grandaor commented 7 months ago

yeah !!!

you're amazing @immauss :)

my openvas is up with traefik now

thanks for your help !

mnaismith commented 7 months ago

Fantastic ! Thanks. Legend. All is well once again.

Matt

From: grandaor @.> Sent: Thursday, November 23, 2023 12:21:55 AM To: immauss/openvas @.> Cc: Matt Naismith @.>; Mention @.> Subject: Re: [immauss/openvas] [BUG] Healthchecks report no issue but container remains unhealthy (Issue #234)

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

yeah !!!

you're amazing @immausshttps://github.com/immauss :)

my openvas is up with traefik now

thanks for your help !

— Reply to this email directly, view it on GitHubhttps://github.com/immauss/openvas/issues/234#issuecomment-1822865092, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A7WQHX3GXQ6NNI7E4JTHYELYFYDAHAVCNFSM6AAAAAA7SNJ7Z6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRSHA3DKMBZGI. You are receiving this because you were mentioned.Message ID: @.***>

immauss commented 7 months ago

Thanks @mnaismith & @grandaor

-Scott