search

immauss / openvas

Containers for running the Greenbone Vulnerability Manager. Run as a single container with all services or separate single applications containers via docker-compose.
GNU Affero General Public License v3.0
368 stars 102 forks source link

using latest amd64 image -> warnings and no reports #44

Closed cybermcm closed 3 years ago

cybermcm commented 3 years ago

Using latest amd64 tag from docker hub with a clean data path (from scratch) sha256:34cc22d839eb28798681285a25c2eb0f1b7c955779c8c0154c7fd68a97a52454

openvas:
  container_name: openvas
  cpus: 4
  environment:
   - PASSWORD=${openvas_OV_PASSWORD}
   - RELAYHOST=${EMAIL_HOST} # immauss/openvas, securecompliance/gvm
   - SMTPPORT=25 # immauss/openvas, securecompliance/gvm
   - QUIET=true # immauss/openvas
   - SKIPSYNC=true
  hostname: openvas.${HOSTNAME}
  image: immauss/openvas
  labels:
   - ${WATCHTOWER_TRUE}
   - diun.enable=true
   - diun.watch_repo=true
   - traefik.enable=true
   - "traefik.http.routers.openvas.rule=Host(`openvas.${HOSTNAME}`)"
   - traefik.http.routers.openvas.entrypoints=web-secure
   - traefik.http.routers.openvas.tls.certresolver=le
   - traefik.http.routers.openvas.tls=true
   - traefik.http.routers.openvas.priority=2
   - traefik.http.routers.openvas.tls.options=default
   - traefik.http.routers.openvas.service=openvas
   - traefik.http.services.openvas.loadbalancer.server.port=9392
  networks:
   - web
  restart: always
  volumes:
   - ${DOCKER_PATH}/openvas:/data # immauss/openvas, securecompliance/gvm

1) seeing a WARNING message every 10 seconds: md manage:WARNING:2021-05-31 20h21.33 utc:32750: secinfo_feed_version_status: last scap database update later than last feed update

2) pdf report not working, download ends with a 0 byte file (correct file name but no content); doesn't work as email report

immauss commented 3 years ago

I think something went wrong with the last rebuild. Marking this as a dupe of #43 for now.

immauss commented 3 years ago

Well .... as it turns out ... the refresh process needs about 5G of space to pull all the feeds, rebuild the database and then compress the pieces. The machine running that ... did not have 5G of space. So it failed out and made a mess. Please give the "latest" a another shot and let me know if you have any issues.

-Scott

cybermcm commented 3 years ago

fresh latest image, still some points:

  1. WARNING messages -> gone -> OK
  2. PDF Report -> still not working -> log: run_report_format_script: No generate script found at /usr/local/var/lib/gvm/gvmd/report_formats/8282ba3d-66e8-4971-859b-745b76a3a65e/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
  3. still strange (worked with SQL 11 as far as I remember), "My settings" sometimes empty Snipaste_2021-06-02_08-43-12 for example, "Default report format" empty, "Default port list" empty, ..... Clipboard01 sometimes this works, most of the time not, any idea?
immauss commented 3 years ago

Are you coming from an old DB, or starting fresh?

I'm not seeing this on a fresh pull and start of "latest" or in my production which is running the latest as well.

Anything interesting in "docker logs" ?

immauss commented 3 years ago

Hmm ... take a look at Lozio's issue ....

cybermcm commented 3 years ago

started with a "fresh" image, no persistent data on startup issue https://github.com/immauss/openvas/issues/43 also happens for my setup: GVMD_DATA 30 days old (but SKIPSYNC startup in my case) He added that his container is unhealthy, mine is healthy

my log:

openvas                     | 8:C 02 Jun 2021 18:03:34.534 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
openvas                     | 8:C 02 Jun 2021 18:03:34.534 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=8, just started
openvas                     | 8:C 02 Jun 2021 18:03:34.534 # Configuration loaded
openvas                     | Wait for redis socket to be created...
openvas                     | Testing redis status...
openvas                     | Redis ready.
openvas                     | Fixing Database folder...
openvas                     | Fixing local/var/lib ...
openvas                     | Fixing local/share ...
openvas                     | Fixing log directory for persistent logs ....
openvas                     | Creating postgresql.conf and pg_hba.conf
openvas                     | Starting PostgreSQL...
openvas                     | waiting for server to start....2021-06-02 16:03:39.015 GMT [30] LOG:  starting PostgreSQL 12.7 (Ubuntu 12.7-1.pgdg20.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, 64-bit
openvas                     | 2021-06-02 16:03:39.016 GMT [30] LOG:  listening on IPv4 address "0.0.0.0", port 5432
openvas                     | 2021-06-02 16:03:39.016 GMT [30] LOG:  listening on IPv6 address "::", port 5432
openvas                     | 2021-06-02 16:03:39.017 GMT [30] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
openvas                     | 2021-06-02 16:03:39.023 GMT [30] LOG:  could not open usermap file "/data/database/pg_ident.conf": No such file or directory
openvas                     | 2021-06-02 16:03:39.025 GMT [31] LOG:  database system was shut down at 2021-06-02 16:03:32 GMT
openvas                     | 2021-06-02 16:03:39.030 GMT [30] LOG:  database system is ready to accept connections
openvas                     |  done
openvas                     | server started
openvas                     | Running first start configuration...
openvas                     | Adding gvm user
openvas                     | Fixing feed rsync options
openvas                     | NOTICE:  relation "vt_severities" already exists, skipping
openvas                     | NOTICE:  relation "vt_severities" already exists, skipping
openvas                     | NOTICE:  relation "vt_severities" already exists, skipping
openvas                     | Migrating the database to the latest version if needed.
openvas                     | Starting Greenbone Vulnerability Manager...
openvas                     | Waiting for gvmd
openvas                     | Waiting for gvmd
openvas                     | admin
openvas                     | Time to fixup the gvm accounts.
openvas                     | Setting admin password
openvas                     | reset
openvas                     | Starting Postfix for report delivery by email
openvas                     |  * Starting Postfix Mail Transport Agent postfix
openvas                     |    ...done.
openvas                     | Starting Open Scanner Protocol daemon for OpenVAS...
openvas                     | Fixing the ospd socket ...
openvas                     | Starting Greenbone Security Assistant...
openvas                     | Oops, secure memory pool already initialized
openvas                     | ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
openvas                     | + Your GVM/openvas/postgresql container is now ready to use! +
openvas                     | ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
openvas                     |
openvas                     | gvmd --version
openvas                     | Greenbone Vulnerability Manager 21.4.0
openvas                     | Manager DB revision 242
openvas                     | Copyright (C) 2009-2021 Greenbone Networks GmbH
openvas                     | License: AGPL-3.0-or-later
openvas                     | This is free software: you are free to change and redistribute it.
openvas                     | There is NO WARRANTY, to the extent permitted by law.
openvas                     |
openvas                     | ++++++++++++++++
openvas                     | + Tailing logs +
openvas                     | ++++++++++++++++
openvas                     | ==> /usr/local/var/log/gvm/gsad.log <==
openvas                     | gsad main:MESSAGE:2021-06-02 05h45.36 utc:923: Starting GSAD version 21.04.0
openvas                     | gsad  gmp:MESSAGE:2021-06-02 06h12.12 utc:925: Authentication success for 'admin' from 11.111.11.11
openvas                     | gsad main:MESSAGE:2021-06-02 16h03.49 utc:767: Starting GSAD version 21.04.0
openvas                     |
openvas                     | ==> /usr/local/var/log/gvm/gvmd.log <==
openvas                     | md   main:WARNING:2021-06-02 16h03.41 utc:63: gvmd: databases are already at the supported version
openvas                     | md   main:MESSAGE:2021-06-02 16h03.41 utc:69:    Greenbone Vulnerability Manager version 21.4.0 (DB revision 242)
openvas                     | md   main:MESSAGE:2021-06-02 16h03.41 utc:74:    Greenbone Vulnerability Manager version 21.4.0 (DB revision 242)
openvas                     | md   main:WARNING:2021-06-02 16h03.41 utc:74: gvmd: Another process is busy starting up
openvas                     | md   main:MESSAGE:2021-06-02 16h03.42 utc:94:    Greenbone Vulnerability Manager version 21.4.0 (DB revision 242)
openvas                     | md   main:WARNING:2021-06-02 16h03.42 utc:94: gvmd: Another process is busy starting up
openvas                     | md   main:MESSAGE:2021-06-02 16h03.43 utc:101:    Greenbone Vulnerability Manager version 21.4.0 (DB revision 242)
openvas                     | md manage:   INFO:2021-06-02 16h03.43 utc:101:    Getting users.
openvas                     | md   main:MESSAGE:2021-06-02 16h03.43 utc:106:    Greenbone Vulnerability Manager version 21.4.0 (DB revision 242)
openvas                     | md manage:   INFO:2021-06-02 16h03.43 utc:106:    Modifying user password.
openvas                     |
openvas                     | ==> /usr/local/var/log/gvm/openvas.log <==
openvas                     | lib  nvticache:MESSAGE:2021-06-02 05h46.57 utc:933: Updated NVT cache from version 0 to 202106011036
openvas                     | sd   main:MESSAGE:2021-06-02 06h18.14 utc:3093: openvas 21.4.0 started
openvas                     | sd   main:MESSAGE:2021-06-02 06h18.17 utc:3093: Vulnerability scan 8e6d1e69-98e1-488a-bc82-336c150ef182 started: Target has 1 hosts: xxx.xxx.xx, with max_hosts = 20 and max_checks = 4
openvas                     | sd   main:MESSAGE:2021-06-02 06h18.17 utc:3101: Vulnerability scan 8e6d1e69-98e1-488a-bc82-336c150ef182 started for host: 80.110.40.138 (Vhosts: x.xxx.xx)
openvas                     | sd   main:MESSAGE:2021-06-02 07h32.06 utc:3101: Vulnerability scan 8e6d1e69-98e1-488a-bc82-336c150ef182 finished for host 80.110.40.138 in 4428.30 seconds
openvas                     | sd   main:MESSAGE:2021-06-02 07h32.06 utc:3093: Vulnerability scan 8e6d1e69-98e1-488a-bc82-336c150ef182 finished in 4432 seconds: 1 hosts
openvas                     |
openvas                     | ==> /usr/local/var/log/gvm/ospd-openvas.log <==
openvas                     | OSPD[906] 2021-06-02 07:45:36,846: INFO: (ospd.main) Starting OSPd OpenVAS version 21.4.0.
openvas                     | OSPD[906] 2021-06-02 08:18:02,183: INFO: (ospd.command.command) Scan 8e6d1e69-98e1-488a-bc82-336c150ef182 added to the queue in position 1.
openvas                     | OSPD[906] 2021-06-02 08:18:03,609: INFO: (ospd.ospd) Currently 1 queued scans.
openvas                     | OSPD[906] 2021-06-02 08:18:03,629: INFO: (ospd.ospd) Starting scan 8e6d1e69-98e1-488a-bc82-336c150ef182.
openvas                     | OSPD[906] 2021-06-02 09:32:07,229: INFO: (ospd.ospd) 8e6d1e69-98e1-488a-bc82-336c150ef182: Host scan finished.
openvas                     | OSPD[906] 2021-06-02 09:32:07,231: INFO: (ospd.ospd) 8e6d1e69-98e1-488a-bc82-336c150ef182: Scan finished.
openvas                     | OSPD[750] 2021-06-02 18:03:49,106: INFO: (ospd.main) Starting OSPd OpenVAS version 21.4.0.
openvas                     |
openvas                     | ==> /usr/local/var/log/gvm/openvas.log <==
openvas                     | lib  nvticache:MESSAGE:2021-06-02 16h04.57 utc:779: Updated NVT cache from version 0 to 202106011036
openvas                     |
openvas                     | ==> /usr/local/var/log/gvm/gvmd.log <==
openvas                     | md manage:WARNING:2021-06-02 18h51.36 CEST:2206: run_report_format_script: No generate script found at /usr/local/var/lib/gvm/gvmd/report_formats/8282ba3d-66e8-4971-859b-745b76a3a65e/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
openvas                     |
openvas                     |
openvas                     | ==> /usr/local/var/log/gvm/gvmd.log <==
openvas                     | md manage:WARNING:2021-06-02 20h00.43 CEST:3999: run_report_format_script: No generate script found at /usr/local/var/lib/gvm/gvmd/report_formats/8282ba3d-66e8-4971-859b-745b76a3a65e/c402cc3e-b531-11e1-9163-406186ea4fc5/generate
immauss commented 3 years ago

So the GVMD_DATA feed was normal. https://community.greenbone.net/t/gvmd-data-too-old-34-days-please-check-the-automatic-synchronization-of-your-system/7565/10

immauss commented 3 years ago

Are you still seeing issues?

cybermcm commented 3 years ago

using latest image with sha256:6a9a71b3882cb7d4d529e5057b3a8ead2acb3f2d66d02b721a98f0bdde9ffb55 2 things: 1) still no reports (tried pdf and csv), log: md manage:WARNING:2021-06-03 17h26.43 CEST:32516: run_report_format_script: No generate script found at /usr/local/var/lib/gvm/gvmd/report_formats/8282ba3d-66e8-4971-859b-745b76a3a65e/c1645568-627a-11e3-a660-406186ea4fc5/generate 2) still sometimes settings are empty. Currently working, didn't change a thing since yesterday. Very weird. But this happens for a long time now, I think since switching to the new POSTGRES db version

immauss commented 3 years ago

OK ... I think there is a regression going on here.... I "thought" I fixed this once before .... now I just need to remember how I fixed it ... I think I have a path getting created wrong, or not being created ... But there is definitely something missing for reports.

immauss commented 3 years ago

OK finally remembered what I had to do last time this came up. I rebuilt the DB from scratch.

Can you give the current "latest" image a try and see if it is still giving you problems.

Thanks, Scott

cybermcm commented 3 years ago

you mentioned the database rebuild here last time, https://github.com/immauss/openvas/issues/31#issuecomment-832472381 ;-) I just tried latest sha256:6a9a71b3882cb7d4d529e5057b3a8ead2acb3f2d66d02b721a98f0bdde9ffb55 clean volumes, from scratch, still no reports: md manage:WARNING:2021-06-07 08h57.45 CEST:2327: run_report_format_script: No generate script found at /usr/local/var/lib/gvm/gvmd/report_formats/8282ba3d-66e8-4971-859b-745b76a3a65e/c402cc3e-b531-11e1-9163-406186ea4fc5/generate

cybermcm commented 3 years ago

@immauss: Any news on this? Do you need any additional logs? Just drop a note if I can do anything...

immauss commented 3 years ago

Sorry ... I've been swamped the last few weeks.

Let's start at the beginning. I'm still not sure why this isn't working for you. I've spun up a new image on multiple platforms, and the reports seems to work.

Can you send me your compose file or how you are starting?

I know .. that shoulnd't make a difference, but I'm grasping at straws to be honest.

and a 'docker system info' output and the full output of 'docker logs' for the container.

Maybe looking at them will trigger something for me.

Thanks, -Scott

cybermcm commented 3 years ago

Scott, I've to apologize... I started again from scratch, this time reports work fine. I don't know what I've missed last time, probably forgot to delete the old image files and pull new ones.

At least I can contribute to the other error I mentioned here, settings won't show correctly.

This is due to my setup with a Traefik reverse proxy. It seems that this causes 502 HTTP errors. Some sort of timing issue. I didn't find a solution until now, but I'll keep looking for it.

I'm closing this issue since your image is working, thanks again for your patience and your time.