[Enhancement] Migration from official securecompliance/gvm image

[landonstewart]

Describe the solution you'd like Many folks, including myself, are using the official but outdated image securecompliance/gvm. It would be nice to be able to switch from that image to this more actively developed one.

When using deineagenturug/gvm:latest-data-full as a drop in replacement for securecompliance/gvm:debian-master-data there are problems with PostgreSQL. Likely because of the version differences.

A check on first start and migration would be really excellent. Failing that - a method to migrate existing data would be great.

ALSO - I'm capable and willing to clone and send pull requests if you can give me a basic overview of how the check/migration should work. At this point its not clear and I'd need to get up to speed before attempting to contribute changes.

[markdesilva]

@markdesilva did you pull the latest image version? It sounds like you did not. I had this error only in old versions and it should already fixed.

Yes I removed all other images and pulled the latest:

REPOSITORY                            TAG       IMAGE ID       CREATED        SIZE
deineagenturug/pgdb-upgrade-develop   latest    c8c05a59b9c4   11 hours ago   676MB

EDIT: Also the path of the lib is different... Which image you used before from SCS?

Was using debian-master. If the path is wrong, maybe I try looking for the proper library path and soft link before running the script. I will try and report back.

Thank you!

[markdesilva]

If the path is wrong, maybe I try looking for the proper library path and soft link before running the script. I will try and report back.

Success! I copied the contents of /usr/local/lib to /usr/lib, then I soft linked /usr/lib in /usr/local

cd /usr/local/lib
cp -R * /usr/lib
cd /usr/local
mv lib lib.1
ln -s /usr/lib .

After running the db_upgrade script, it completes without errors!

[Dexus]

THX @netbix @markdesilva you help a lot to improve the db_upgrade script. ❤️

It only need /usr/lib/libgvm-pg-server symlinked, but I will look how I can find wrong locations and re-set them to be correct. But I would like to prevent the symlinks, because it will make problems and need changes for the gvm container, which I will prevent.

I'm not so familiar with the postgres DB... I know that some functions that are created by gvmd have the location to /usr/lib/libgvm-pg-server or in your error the /usr/local/lib/libgvm-pg-server as path. To make sure that this will get fixed we have to run there a fix to set the correct path to the libgvm-pg-server else you will have problems with the gvm container, without modifications.

[Dexus]

Here a what I found, that use the libgvm-pg-server in database gvmd

CREATE OR REPLACE FUNCTION public.hosts_contains(
    text,
    text)
    RETURNS boolean
    LANGUAGE 'c'
    COST 1
    IMMUTABLE PARALLEL UNSAFE
AS '/usr/lib/libgvm-pg-server', 'sql_hosts_contains'
;

ALTER FUNCTION public.hosts_contains(text, text)
    OWNER TO dba;
CREATE OR REPLACE FUNCTION public.max_hosts(
    text,
    text)
    RETURNS integer
    LANGUAGE 'c'
    COST 1
    VOLATILE PARALLEL UNSAFE
AS '/usr/lib/libgvm-pg-server', 'sql_max_hosts'
;

ALTER FUNCTION public.max_hosts(text, text)
    OWNER TO dba;
CREATE OR REPLACE FUNCTION public.next_time_ical(
    text,
    text)
    RETURNS integer
    LANGUAGE 'c'
    COST 1
    VOLATILE PARALLEL UNSAFE
AS '/usr/lib/libgvm-pg-server', 'sql_next_time_ical'
;

ALTER FUNCTION public.next_time_ical(text, text)
    OWNER TO dba;
CREATE OR REPLACE FUNCTION public.next_time_ical(
    text,
    text,
    integer)
    RETURNS integer
    LANGUAGE 'c'
    COST 1
    VOLATILE PARALLEL UNSAFE
AS '/usr/lib/libgvm-pg-server', 'sql_next_time_ical'
;

ALTER FUNCTION public.next_time_ical(text, text, integer)
    OWNER TO dba;

CREATE OR REPLACE FUNCTION public.regexp(
    text,
    text)
    RETURNS boolean
    LANGUAGE 'c'
    COST 1
    VOLATILE PARALLEL UNSAFE
AS '/usr/lib/libgvm-pg-server', 'sql_regexp'
;

ALTER FUNCTION public.regexp(text, text)
    OWNER TO dba;

[Dexus]

@markdesilva @netbix Thanks for your support here and testing.

New Version is ready.

docker pull deineagenturug/pgdb-upgrade-develop:latest

Fixed:

• correct old libgvm-pg-server path
• fix rm to remove also DOT files from the new DB directory

[markdesilva]

THX @netbix @markdesilva you help a lot to improve the db_upgrade script. ❤️

Happy to help with my limited knowledge!

I'm not so familiar with the postgres DB... I know that some functions that are created by gvmd have the location to /usr/lib/libgvm-pg-server or in your error the /usr/local/lib/libgvm-pg-server as path. To make sure that this will get fixed we have to run there a fix to set the correct path to the libgvm-pg-server else you will have problems with the gvm container, without modifications.

Yes, I had to do the same softlinking when I started the gvm container.

Also when I run the latest gvm container with this upgraded db, I'm getting the error that @netbix was encountering:

createuser: error: creation of new role failed: ERROR: role "gvm" already exists

But I noticed that .firstrun and .upgrade_to_21.4.0 were not in the upgraded database folder. I touched the files in the database folder and it works from there.

Additionally I see this, which I'm not sure is normal:

2022-04-24 17:47:06,062 INFO spawned: 'rsyslogd' with pid 573 2022-04-24 17:47:06,070 INFO exited: rsyslogd (exit status 1; not expected) 2022-04-24 17:47:07,072 INFO gave up: rsyslogd entered FATAL state, too many start retries too quickly

Checking the processes, rsyslogd is not running.

For upgrading db from a previous gvm to a current one, I think we must have the dot files

/var/lib/gvm/.created_gvm_user (In host ./storage/gvm/.created_gvm_user) /opt/database/.firstrun (In host ./storage/database/.firstrun) /opt/database/.upgrade_to_21.4.0 (In host ./storage/database/.upgrade_to_21.4.0)

Then the gvm container runs smooth.

My gvm is now up and running, my reports, tagest, overrides, tasks, etc are all there!

GVMUpdate: started
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+     Greenbone Vulnerability Manager 21.4.5
Manager DB revision 242
Copyright (C) 2009-2021 Greenbone Networks GmbH
License: AGPL-3.0-or-later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ Your GVM container is now ready to use!                 +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Testing the scanning now.

Thank you @Dexus!

[Dexus]

THX @netbix @markdesilva you help a lot to improve the db_upgrade script. ❤️

Happy to help with my limited knowledge!

I'm not so familiar with the postgres DB... I know that some functions that are created by gvmd have the location to /usr/lib/libgvm-pg-server or in your error the /usr/local/lib/libgvm-pg-server as path. To make sure that this will get fixed we have to run there a fix to set the correct path to the libgvm-pg-server else you will have problems with the gvm container, without modifications.

Yes, I had to do the same softlinking when I started the gvm container.

Also when I run the latest gvm container with this upgraded db, I'm getting the error that @netbix was encountering:

createuser: error: creation of new role failed: ERROR: role "gvm" already exists

But I noticed that .firstrun and .upgrade_to_21.4.0 were not in the upgraded database folder. I touched the files in the database folder and it works from there.

Additionally I see this, which I'm not sure is normal:

2022-04-24 17:47:06,062 INFO spawned: 'rsyslogd' with pid 573 2022-04-24 17:47:06,070 INFO exited: rsyslogd (exit status 1; not expected) 2022-04-24 17:47:07,072 INFO gave up: rsyslogd entered FATAL state, too many start retries too quickly

Checking the processes, rsyslogd is not running.

For upgrading db from a previous gvm to a current one, I think we must have the dot files

/var/lib/gvm/.created_gvm_user (In host ./storage/gvm/.created_gvm_user) /opt/database/.firstrun (In host ./storage/database/.firstrun) /opt/database/.upgrade_to_21.4.0 (In host ./storage/database/.upgrade_to_21.4.0)

Then the gvm container runs smooth.

My gvm is now up and running, my reports, tagest, overrides, tasks, etc are all there!

GVMUpdate: started
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+     Greenbone Vulnerability Manager 21.4.5
Manager DB revision 242
Copyright (C) 2009-2021 Greenbone Networks GmbH
License: AGPL-3.0-or-later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ Your GVM container is now ready to use!                 +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Testing the scanning now.

Thank you @Dexus!

This should all gone when you run this again with the old db. Also the database should get the corrected path to the libs in the new db.

If you get then again errors let me know. Maybe the gvmd.log in /var/log/gvm/ will print errors or the postgres logs if there after the db upgrade anything wrong that is not yet in line with the gvm container. Which I hope is not the thing.

[Dexus]

One thing: rsyslog (#12) error is already in pipeline > deinegenturug/gvm-develop:latest

[markdesilva]

Hi @Dexus, I ran the latest deineagenturug/pgdb-upgrade-develop:latest on my old db and it fails with:

Start checking of Postgresql pg_upgrade from 12 to 13
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for presence of required libraries                 fatal

Your installation references loadable libraries that are missing from the
new installation.  You can add these libraries to the new installation,
or remove the functions using them from the old installation.  A list of
problem libraries is in the file:
    loadable_libraries.txt

Failure, exiting
root@dc2dd6914f3a:/work# more loadable_libraries.txt
could not load library "/usr/local/lib/libgvm-pg-server": ERROR:  could not access file "/usr/local/lib/libgvm-pg-server": No such file or directory
In database: gvmd

Have confirmed I'm running the latest deineagenturug/pgdb-upgrade-develop:

REPOSITORY                            TAG           IMAGE ID       CREATED       SIZE
deineagenturug/pgdb-upgrade-develop   latest        02275e667d97   6 hours ago   676MB

@markdesilva @netbix Thanks for your support here and testing.

New Version is ready.

docker pull deineagenturug/pgdb-upgrade-develop:latest

Fixed:

* correct old `libgvm-pg-server` path

* fix rm to remove also DOT files from the new DB directory

[Dexus]

@markdesilva can you please send me a copy from your old DB as a .tar.gz/xz ? https://nc.josef-froehle.de/s/GmMW3txfAW8mCpY I can so test it local and I would be a bit faster to fix problems. And don't worry, I will work with it under data protection aspects.As soon as I have done my tests, I delete everything again.

[markdesilva]

@markdesilva can you please send me a copy from your old DB as a .tar.gz/xz ? https://nc.josef-froehle.de/s/GmMW3txfAW8mCpY I can so test it local and I would be a bit faster to fix problems. And don't worry, I will work with it under data protection aspects.As soon as I have done my tests, I delete everything again.

Sure, the whole thing is 4.3GB though, not sure if it'll go through. Let me try.

[Dexus]

Sure, the whole thing is 4.3GB though, not sure if it'll go through. Let me try.

Should not be a problem for my Server, if you have an alternate storage where I can download it, I can do it. Then you can send me a link to github@josef-froehle.de

[markdesilva]

Sure, the whole thing is 4.3GB though, not sure if it'll go through. Let me try.

Should not be a problem for my Server, if you have an alternate storage where I can download it, I can do it. Then you can send me a link to github@josef-froehle.de

After compression its a few hundred MB. Uploading now.

[markdesilva]

After compression its a few hundred MB. Uploading now.

Upload is complete. Thanks!

[Dexus]

After compression its a few hundred MB. Uploading now.

Upload is complete. Thanks!

Hmm, nothing there. But got an error in the logs. Forgot to set the Upload limit in one config file which now is fixed. Can you re-try?

Edit: @markdesilva thank you upload is there now.

[markdesilva]

Hmm, nothing there. But got an error in the logs. Forgot to set the Upload limit in one config file which now is fixed. Can you re-try?

Done! Thanks.

[Dexus]

New Version is ready.

docker pull deineagenturug/pgdb-upgrade-develop:latest

Fixed:

• correct path of the /usr/lib/libgvm-pg-server.so symlink to /usr/local/lib/libgvm-pg-server.so now with the extension