20.8 does not populate default scan configs upon database initialization

mcjon3z commented 4 years ago

Expected behavior

Upon launching a clean instance with no previous database, GVMD should populate port lists, report formats, and scan configs from the data feed sync.

Actual behavior

All feeds are properly synced prior to launching GVMD. Events in the log show report formats and port lists are created, however there are no log entries (success or failure) relating to scan configs. Have verified that the scan config files are synced and stored in /usr/local/var/lib/gvm/data-objects/gvmd/20.08/configs. Also have verified that permissions are correct and the same as the permissions for port lists and report formats.

Steps to reproduce

Clean installation of all 20.8 components from source on Ubuntu 18.4
Run feed syncs for GVMD_DATA, SCAP, and CERT
Create admin user and grant feed import rights
Launch GVMD

GVM versions

gsa: 20.08.0

gvm: 20.08.0

openvas-scanner: 20.08.0

gvm-libs: 20.08.0

Environment

Operating system:

Linux gvm 4.19.76-linuxkit #1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

Installation method / source: source installation

Logfiles

md   main:MESSAGE:2020-08-14 16h08.53 utc:590:    Greenbone Vulnerability Manager version 20.08.0 (DB revision 233)
md   main:   INFO:2020-08-14 16h08.53 utc:590:    Migrating database.
md   main:WARNING:2020-08-14 16h08.53 utc:590: manage_migrate: no task tables yet, so no need to migrate them
md   main:MESSAGE:2020-08-14 16h08.53 utc:590: No SCAP database found for migration
md   main:MESSAGE:2020-08-14 16h08.53 utc:590: No CERT database found for migration
md   main:WARNING:2020-08-14 16h08.53 utc:590: gvmd: databases are already at the supported version
md   main:MESSAGE:2020-08-14 16h08.53 utc:593:    Greenbone Vulnerability Manager version 20.08.0 (DB revision 233)
md manage:   INFO:2020-08-14 16h08.53 utc:593:    Creating user.
md manage:WARNING:2020-08-14 16h08.53 utc:593: sql_exec_internal: PQexec failed: ERROR:  relation "public.meta" does not exist
LINE 1: SELECT value FROM public.meta WHERE name = 'database_version...
                          ^
 (7)
md manage:WARNING:2020-08-14 16h08.53 utc:593: sql_exec_internal: SQL: SELECT value FROM public.meta WHERE name = 'database_version';
md manage:WARNING:2020-08-14 16h08.53 utc:593: sql_x: sql_exec_internal failed
md manage:MESSAGE:2020-08-14 16h08.53 utc:593: No SCAP database found
md manage:MESSAGE:2020-08-14 16h08.53 utc:593: No CERT database found
md   main:MESSAGE:2020-08-14 16h08.54 utc:596:    Greenbone Vulnerability Manager version 20.08.0 (DB revision 233)
md manage:   INFO:2020-08-14 16h08.54 utc:596:    Modifying setting.
md manage:MESSAGE:2020-08-14 16h08.54 utc:596: No SCAP database found
md manage:MESSAGE:2020-08-14 16h08.54 utc:596: No CERT database found
md   main:MESSAGE:2020-08-14 16h08.54 utc:600:    Greenbone Vulnerability Manager version 20.08.0 (DB revision 233)
md manage:   INFO:2020-08-14 16h08.54 utc:600:    Getting users.
md manage:MESSAGE:2020-08-14 16h08.54 utc:600: No SCAP database found
md manage:MESSAGE:2020-08-14 16h08.54 utc:600: No CERT database found
md   main:MESSAGE:2020-08-14 16h08.54 utc:605:    Greenbone Vulnerability Manager version 20.08.0 (DB revision 233)
md manage:   INFO:2020-08-14 16h08.54 utc:605:    Modifying setting.
md manage:MESSAGE:2020-08-14 16h08.54 utc:605: No SCAP database found
md manage:MESSAGE:2020-08-14 16h08.54 utc:605: No CERT database found
md   main:MESSAGE:2020-08-14 16h08.54 utc:540:    Greenbone Vulnerability Manager version 20.08.0 (DB revision 233)
md manage:MESSAGE:2020-08-14 16h08.54 utc:540: No SCAP database found
md manage:MESSAGE:2020-08-14 16h08.54 utc:540: No CERT database found
event port_list:MESSAGE:2020-08-14 16h08.55 utc:540: Port list All IANA assigned TCP (33d0cd82-57c6-11e1-8ed1-406186ea4fc5) has been created by admin
event port_list:MESSAGE:2020-08-14 16h08.55 utc:540: Port list All TCP and Nmap top 100 UDP (730ef368-57e2-11e1-a90f-406186ea4fc5) has been created by admin
event port_list:MESSAGE:2020-08-14 16h08.55 utc:540: Port list All IANA assigned TCP and UDP (4a4717fe-57d2-11e1-9a26-406186ea4fc5) has been created by admin
event report_format:MESSAGE:2020-08-14 16h08.55 utc:540: Report format ITG (77bd6c4a-1f62-11e1-abf0-406186ea4fc5) has been created by admin
event report_format:MESSAGE:2020-08-14 16h08.55 utc:540: Report format TXT (a3810a62-1f62-11e1-9219-406186ea4fc5) has been created by admin
event report_format:MESSAGE:2020-08-14 16h08.55 utc:540: Report format CSV Results (c1645568-627a-11e3-a660-406186ea4fc5) has been created by admin
event report_format:MESSAGE:2020-08-14 16h08.55 utc:540: Report format PDF (c402cc3e-b531-11e1-9163-406186ea4fc5) has been created by admin
event report_format:MESSAGE:2020-08-14 16h08.55 utc:540: Report format XML (a994b278-1f62-11e1-96ac-406186ea4fc5) has been created by admin
event report_format:MESSAGE:2020-08-14 16h08.55 utc:540: Report format Anonymous XML (5057e5cc-b825-11e4-9d0e-28d24461215b) has been created by admin
md manage:   INFO:2020-08-14 16h10.25 utc:637: Initializing CERT database
md manage:WARNING:2020-08-14 16h10.25 utc:635: update_scap: No SCAP db present, rebuilding SCAP db from scratch
md manage:   INFO:2020-08-14 16h10.25 utc:637: sync_cert: Updating data from feed

bjoernricks commented 4 years ago

Hi,

a Scan Config relies on available NVTs. Therefore you need to connect an ospd-openvas scanner to gvmd to sync the nvts. If gvmd has no nvt data it can't handle the scan configs.

mcjon3z commented 4 years ago

NVT is synced as well and GVMD connects to the ospd daemon. Did not include that in my original post. I will check the launch order on my startup scripts tomorrow and see if there may be an issue there.

bjoernricks commented 4 years ago

ok. Did you configure the Feed Import Owner https://github.com/greenbone/gvmd/blob/gvmd-20.08/INSTALL.md#set-the-feed-import-owner ?

mcjon3z commented 4 years ago

Upon further investigation, it appears that the default openvas scanner socket that gvmd is looking for is no longer at /tmp/ospd.sock, but rather /var/run/ospd/ospd.sock - thus, it is not importing the NVT's from openvas as you mentioned before.

mcjon3z commented 4 years ago

Correcting the socket path that GVMD looks for allowed the NVT's to be imported and scan configs to get created. Thank you for your quick response - I'm going ahead and closing this issue.

cfi-gb commented 4 years ago

@mcjon3z Thanks for posting this info. The default OSPD socket path was indeed changed as documented in the CHANGELOG.md;

Set the default OSPD unix socket path to /var/run/ospd/ospd.sock #1238

which is also reflected in the INSTALL.md here:

https://github.com/greenbone/gvmd/blob/v20.8.0/INSTALL.md#configure-the-default-ospd-scanner-socket-path

I have updated https://community.greenbone.net/t/gvm-20-08-stable-initial-release-2020-08-12/6312 now to include this info to make it a little bit more visible.

mcjon3z commented 4 years ago

@cfi-gb - good call. I did find the changed INSTALL.md path when I started searching for the line in the source code referencing the socket path and trying to figure out the best way to handle the change in my docker builds. The modification to the release notes should point others to the change. This was really one of only two gotchas that I encountered in migrating to 20.8, the other being the requirement for the version 11 report files to be present to upgrade the database which was also discussed in the release notes.

New version seems to be running well and loads faster upon startup. I'll probably give it a week or two more of testing, but it seems to be production ready so far.

bjoernricks commented 4 years ago

@mcjon3z thanks for the feedback! Just let us know about possible additional issues.

greenbone / gvmd