yu210148 / gvm_install

A script to install GVM 20 or 21 on Ubuntu 20.04 or Debian 10.
GNU General Public License v3.0
142 stars 71 forks source link

Postgres setup fails during installation. #73

Closed chrisnyc closed 2 years ago

chrisnyc commented 2 years ago

Hi, so installing v21 on Ubuntu 20.04 on GCP.

Seems the postgresql setup failed. I think this is because I'm using a low-spec e2-medium VM in GCP and with everything else going on in the script, the machine is bogged down at this point. Below are the logs showing the failures:

running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
    pg_ctlcluster 12 main start
Ver Cluster Port Status Owner    Data directory              Log file
12  main    5432 down   postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log
update-alternatives: using /usr/share/postgresql/12/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode
Setting up sysstat (12.2.0-2ubuntu0.1) ...
..........
Setting up postgresql-server-dev-12 (12.9-0ubuntu0.20.04.1) ...
Setting up postgresql-server-dev-all (214ubuntu0.1) ...
Processing triggers for systemd (245.4-4ubuntu3.16) ...
Processing triggers for libc-bin (2.31-0ubuntu9.7) ...
createuser: error: could not connect to database template1: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
createdb: error: could not connect to database template1: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: error: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: error: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: error: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: error: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Synchronizing state of postgresql.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable postgresql
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   137  100   137    0     0   2107      0 --:--:-- --:--:-- --:--:--  2107
100  291k    0  291k    0     0  2010k      0 --:--:-- --:--:-- --:--:-- 2010k
gvm-libs-21.4.4/
gvm-libs-21.4.4/.clang-format
gvm-libs-21.4.4/.codecov.yml
.........  

UUID=$(/opt/gvm/sbin/gvmd --get-scanners | grep Default | awk '{print $1}')
sleep 10
echo 
/opt/gvm/sbin/gvmd --modify-scanner=$UUID --scanner-host=/opt/gvm/var/run/ospd.sock
sleep 10
/opt/gvm/sbin/gvmd --verify-scanner=$UUID
/opt/gvm/sbin/gvmd --create-user gvmadmin --password=StrongPass
UUID=$(/opt/gvm/sbin/gvmd --get-users --verbose | grep admin | awk '{print $2}')
/opt/gvm/sbin/gvmd --modify-setting 78eceaec-3385-11ea-b237-28d24461215b --value $UUID

(gvmd:21293): md   main-CRITICAL **: 11:38:55.777: gvmd: g_option_context_parse: Missing argument for --value
chrisnyc commented 2 years ago

I solved the problem by manually installing postgresql first with the following commands. Note that I added the postgresql-client. I'm not sure if that helped or not.

apt-get -y install postgresql postgresql-contrib postgresql-server-dev-all postgresql-client
systemctl enable postgresql
systemctl start postgresql

Next, I waited a minute, and then ran systemctl status postgresql

The output should look like this:

● postgresql.service - PostgreSQL RDBMS
     Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)
     Active: active (exited) since Fri 2022-04-15 16:09:22 EDT; 25s ago
   Main PID: 13944 (code=exited, status=0/SUCCESS)
      Tasks: 0 (limit: 4696)
     Memory: 0B
     CGroup: /system.slice/postgresql.service

Apr 15 16:09:22 dev-int-ovas systemd[1]: Starting PostgreSQL RDBMS...
Apr 15 16:09:22 dev-int-ovas systemd[1]: Finished PostgreSQL RDBMS.

Then I commented out the installation steps on the lines 112,113 and 120,121 of install_gvm.sh like this:

#apt-get -y install postgresql postgresql-contrib postgresql-server-dev-all
#systemctl restart postgresql
sudo -Hiu postgres createuser gvm
sudo -Hiu postgres createdb -O gvm gvmd
sudo -Hiu postgres psql -c 'create role dba with superuser noinherit;' gvmd
sudo -Hiu postgres psql -c 'grant dba to gvm;' gvmd
sudo -Hiu postgres psql -c 'create extension "uuid-ossp";' gvmd
sudo -Hiu postgres psql -c 'create extension "pgcrypto";' gvmd
#systemctl restart postgresql
#systemctl enable postgresql

Finally, I ran sudo ./install_gvm.sh -v 21 -u per the regular procedure.

The output of the postgres commands above should look like this:

CREATE ROLE
GRANT ROLE
CREATE EXTENSION
CREATE EXTENSION

Thanks very much for this script! It's very helpful!

yu210148 commented 2 years ago

Hi there,

Thanks for this. Ya, it could be some sort of timing issue. I haven't noticed this in testing but that could be the specs of the machine. Next time I have it open I'll add in the postgresql-client to the list of things that get installed with apt just in case. I don't think it would do any harm to have it in there.

Kev.

On Fri, Apr 15, 2022, 4:34 PM chrisnyc @.***> wrote:

I think this may be occurring because I'm using a lowish-spec e2-medium VM in GCP.

I solved the problem by manually installing postgresql first with the following commands. Note that I added the postgresql-client. I'm not sure if that helped or not.

apt-get -y install postgresql postgresql-contrib postgresql-server-dev-all postgresql-client

systemctl enable postgresql

systemctl start postgresql

Next, I waited a minute, and then ran systemctl status postgresql

The output should look like this:

● postgresql.service - PostgreSQL RDBMS

 Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)

 Active: active (exited) since Fri 2022-04-15 16:09:22 EDT; 25s ago

Main PID: 13944 (code=exited, status=0/SUCCESS)

  Tasks: 0 (limit: 4696)

 Memory: 0B

 CGroup: /system.slice/postgresql.service

Apr 15 16:09:22 dev-int-ovas systemd[1]: Starting PostgreSQL RDBMS...

Apr 15 16:09:22 dev-int-ovas systemd[1]: Finished PostgreSQL RDBMS.

Then I commented out the installation steps on the lines 112,113 and 120,121 of install_gvm.sh like this:

apt-get -y install postgresql postgresql-contrib postgresql-server-dev-all

systemctl restart postgresql

sudo -Hiu postgres createuser gvm

sudo -Hiu postgres createdb -O gvm gvmd

sudo -Hiu postgres psql -c 'create role dba with superuser noinherit;' gvmd

sudo -Hiu postgres psql -c 'grant dba to gvm;' gvmd

sudo -Hiu postgres psql -c 'create extension "uuid-ossp";' gvmd

sudo -Hiu postgres psql -c 'create extension "pgcrypto";' gvmd

systemctl restart postgresql

systemctl enable postgresql

Finally, I ran sudo ./install_gvm.sh -v 21 -u per the regular procedure.

The output of the postgres commands above should look like this:

CREATE ROLE

GRANT ROLE

CREATE EXTENSION

CREATE EXTENSION

Thanks very much for this script! It's very helpful!

— Reply to this email directly, view it on GitHub https://github.com/yu210148/gvm_install/issues/73#issuecomment-1100381408, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKFY4NMIQPJKXWL5TDHCULVFHHDVANCNFSM5TQWZ4UQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

chrisnyc commented 2 years ago

I think trying to install Postgresql while the build is running in the background may be causing the issue.

Running the postgresql installation before you kickoff the build would be one way to solve it with some certainty.

Or possibly, a couple minutes of sleep after the startup of postgresql on line 113, just before the call to "createuser gvm" might give enough time for postgresql to come up before trying to "createuser gvm" would be a simple way to resolve it.

Thanks again for putting this together. Saves me lots of work!

yu210148 commented 2 years ago

Ya, I could throw a sleep statement in there. Will do next time I have it open or if you're looking at it if you could send me a pull request I'll merge it in. I'm not at my screen at the moment.

Kev.

On Fri, Apr 15, 2022, 6:21 PM chrisnyc @.***> wrote:

I think trying to install Postgresql while the build is running in the background may be causing the issue.

Running the postgresql installation before you kickoff the build would be one way to solve it with some certainty.

Or possibly, a couple minutes of sleep after the startup of postgresql on line 113, just before the call to "createuser gvm" might give enough time for postgresql to come up before trying to "createuser gvm" would be a simple way to resolve it.

Thanks again for putting this together. Saves me lots of work!

— Reply to this email directly, view it on GitHub https://github.com/yu210148/gvm_install/issues/73#issuecomment-1100441265, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKFY4JFELFMQXXSWKPMVR3VFHTX3ANCNFSM5TQWZ4UQ . You are receiving this because you commented.Message ID: @.***>

chrisnyc commented 2 years ago

I'll try a few things out and if I find a solid solution I'll send you a pull request. Thanks again!

yu210148 commented 2 years ago

Cool, thanks 👍

Kev.

On Fri, Apr 15, 2022, 6:35 PM chrisnyc @.***> wrote:

I'll try a few things out and if I find a solid solution I'll send you a pull request. Thanks again!

— Reply to this email directly, view it on GitHub https://github.com/yu210148/gvm_install/issues/73#issuecomment-1100448393, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKFY4MAHNH35TDMRMAUIRDVFHVLVANCNFSM5TQWZ4UQ . You are receiving this because you commented.Message ID: @.***>

yu210148 commented 2 years ago

Hi again,

I just added a sleep 60 statement to allow postgres to restart after line 113. Ya, it might be annoying to someone not having this issue but the script already takes a long time to do what it's doing so what's 60 more seconds :)

kev.

chrisnyc commented 2 years ago

Hi, thanks for trying the "sleep 60" after the postgresql install, but it seems waiting alone does not solve the issue. I tested it and here's the result:

Setting up postgresql-server-dev-all (214ubuntu0.1) ...
Processing triggers for systemd (245.4-4ubuntu3.16) ...
Processing triggers for libc-bin (2.31-0ubuntu9.7) ...
Sleeping for a minute to let postgres restart.
https://github.com/yu210148/gvm_install/issues/73 has more info on why we're waiting here.
createuser: error: could not connect to database template1: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
createdb: error: could not connect to database template1: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: error: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: error: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: error: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: error: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Synchronizing state of postgresql.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable postgresql

I will a few more things and let you know what I find. Thanks again for your help.

chrisnyc commented 2 years ago

I'm stumped. I've tried everything I can think of to get Postgres to work immediately after installing it within the script. If I install Postgres in my terraform startup script that runs when the instance is first created, everything works fine. The problem has something to do with the installation of Postgres, and the initialization of the database for first use.

It's possible that I'm seeing this because of CIS server hardening I've got in place, where they require you do things like set /var/log to 'noexec'.

The simple work-around is to just install postgres before you run the script.

yu210148 commented 2 years ago

Odd, okay, thanks for looking into it and documenting a workaround here. Hopefully, anyone else running into the issue will find this thread.

Kev.

On Sun, Apr 17, 2022, 10:30 AM chrisnyc @.***> wrote:

I'm stumped. I've tried everything to get Postgres to work immediately after installing it within the script. If I install Postgres in my terraform startup script that runs when the instance is first created, everything works fine. The problem has something to do with the installation of Postgres, and the initialization of the database for first use.

It's possible that I'm seeing this because of CIS server hardening I've got in place, where they require you do things like set /var/log to 'noexec'.

The simple work-around is to just install postgres before you run the script.

— Reply to this email directly, view it on GitHub https://github.com/yu210148/gvm_install/issues/73#issuecomment-1100890524, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKFY4ME3KMLKLRZFH2OMRTVFQN77ANCNFSM5TQWZ4UQ . You are receiving this because you commented.Message ID: @.***>