Open vchepkov opened 3 months ago
~Duplicate of #413~
@vchepkov If the service fail to restart, there are corrective changes right? What are they?
Also the message indicate that the server cannot reach the database in 3s and timeout. Have you tried to measure how long it take to connect? Maybe some changes delay the connection and a raising the value way workaround the issue?
That's a good question. I only used it for bootstrap and it didn't start. I will try to see, what correction v8.0.1 will do
PuppetDB runs on the same server as database, so, there is no any delay
From what I can tell, only permissions on configuration files are changing and puppetdb is still operational
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: /File[/opt/puppetlabs/puppet/cache/lib/facter/puppetdb_version.rb]/ensure: removed
Info: Loading facts
Notice: Requesting catalog from puppet.localdomain:8140 (192.168.56.20)
Notice: Catalog compiled by puppet.localdomain
Info: Caching catalog for puppet.localdomain
Info: Applying configuration version '076addd'
Notice: /Stage[main]/Puppetdb::Server::Database/File[/etc/puppetlabs/puppetdb/conf.d/database.ini]/owner: owner changed 'root' to 'puppetdb'
Notice: /Stage[main]/Puppetdb::Server::Database/File[/etc/puppetlabs/puppetdb/conf.d/database.ini]/mode: mode changed '0640' to '0600'
Info: Class[Puppetdb::Server::Database]: Scheduling refresh of Service[puppetdb]
Notice: /Stage[main]/Puppetdb::Server::Read_database/File[/etc/puppetlabs/puppetdb/conf.d/read_database.ini]/owner: owner changed 'root' to 'puppetdb'
Notice: /Stage[main]/Puppetdb::Server::Read_database/File[/etc/puppetlabs/puppetdb/conf.d/read_database.ini]/mode: mode changed '0640' to '0600'
Info: Class[Puppetdb::Server::Read_database]: Scheduling refresh of Service[puppetdb]
Notice: /Stage[main]/Puppetdb::Server::Jetty/File[/etc/puppetlabs/puppetdb/conf.d/jetty.ini]/owner: owner changed 'root' to 'puppetdb'
Notice: /Stage[main]/Puppetdb::Server::Jetty/File[/etc/puppetlabs/puppetdb/conf.d/jetty.ini]/mode: mode changed '0640' to '0600'
Info: Class[Puppetdb::Server::Jetty]: Scheduling refresh of Service[puppetdb]
Notice: /Stage[main]/Puppetdb::Server::Puppetdb/File[/etc/puppetlabs/puppetdb/conf.d/puppetdb.ini]/owner: owner changed 'root' to 'puppetdb'
Notice: /Stage[main]/Puppetdb::Server::Puppetdb/File[/etc/puppetlabs/puppetdb/conf.d/puppetdb.ini]/mode: mode changed '0640' to '0600'
Info: Class[Puppetdb::Server::Puppetdb]: Scheduling refresh of Service[puppetdb]
Notice: /Stage[main]/Puppetdb::Server/Service[puppetdb]: Triggered 'refresh' from 4 events
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 12.20 seconds
But if you start with v8.1.0, puppetdb service doesn't start, strange
@smortex , this code has a logic flaw
$puppetdb_major_version = $puppetdb_version ? {
'latest' => '8',
'present' => '8',
default => $puppetdb_version.split('.')[0],
}
Since puppetdb is not installed yet during initial run, puppetdb_version is nil, and split probably provides not the result one would expect
P.S. never mind, it's a variable too
PuppetDB runs on the same server as database, so, there is no any delay
Could be a missing ordering dependency? Configure the database and only then attempt to start PuppetDB that need it?
I am not sure, wasn't able to find a culrpit yet. You can easily reproduce it using a vagrant project: https://github.com/vchepkov/puppet-bootstrap
PUPPET_ENV=puppetdb vagrant up
In case anyone else messes up their system with v8.1.0 of puppetlabs-puppetdb, I was able to recover (on Debian 11) by setting the puppetlabs-puppetdb version to 8.0.1 in Puppetfile
, doing an r10k deploy
, then basically used the procedure here, thanks to @tuxmea: https://groups.google.com/g/puppet-users/c/pc-cHV-0lBY/m/z1CHUTBFBAAJ
sudo systemctl stop puppetdb
sudo aptitude purge puppetdb
sudo mv -vi /etc/puppetlabs/puppetdb /etc/puppetlabs/puppet/{puppetdb.conf,routes.yaml} /tmp
sudo systemctl restart postgresql
sudo puppet agent --test
I have compared the catalogs produced by 8.0.1 and 8.1.0
puppet: Notice: /Stage[main]/Postgresql::Server::Reload/Postgresql::Server::Instance::Reload[main]/Exec[postgresql_reload_main]: Triggered 'refresh' from 1 event
puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Postgresql::Server::Db[puppetdb]/Postgresql::Server::Role[puppetdb]/Postgresql_psql[CREATE ROLE puppetdb ENCRYPTED PASSWORD ****]/command: changed [redacted] to [redacted]
puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Postgresql::Server::Db[puppetdb]/Postgresql::Server::Role[puppetdb]/Postgresql_psql[ALTER ROLE puppetdb ENCRYPTED PASSWORD ****]/command: changed [redacted] to [redacted]
- puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Postgresql::Server::Db[puppetdb]/Postgresql::Server::Database[puppetdb]/Postgresql_psql[CREATE DATABASE "puppetdb"]/command: command changed 'notrun' to 'CREATE DATABASE "puppetdb" WITH TEMPLATE = "template0" '
+ puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Postgresql::Server::Db[puppetdb]/Postgresql::Server::Database[puppetdb]/Postgresql_psql[CREATE DATABASE "puppetdb"]/command: command changed 'notrun' to 'CREATE DATABASE "puppetdb" WITH TEMPLATE = "template0" ENCODING = \'UTF8\' LC_COLLATE = \'en_US.UTF-8\' LC_CTYPE = \'en_US.UTF-8\' '
puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Postgresql::Server::Db[puppetdb]/Postgresql::Server::Database[puppetdb]/Postgresql_psql[REVOKE CONNECT ON DATABASE "puppetdb" FROM public]: Triggered 'refresh' from 1 event
+ puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Postgresql::Server::Db[puppetdb]/Postgresql::Server::Database[puppetdb]/Postgresql_psql[UPDATE pg_database SET datistemplate = false WHERE datname = 'puppetdb']/command: command changed 'notrun' to 'UPDATE pg_database SET datistemplate = false WHERE datname = \'puppetdb\''
puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Postgresql::Server::Db[puppetdb]/Postgresql::Server::Database_grant[GRANT puppetdb - all - puppetdb]/Postgresql::Server::Grant[database:GRANT puppetdb - all - puppetdb]/Postgresql_psql[grant:database:GRANT puppetdb - all - puppetdb]/command: command changed 'notrun' to 'GRANT ALL ON DATABASE "puppetdb" TO "puppetdb"'
puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Postgresql_psql[revoke all access on public schema]/command: command changed 'notrun' to 'REVOKE CREATE ON SCHEMA public FROM public'
puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Postgresql_psql[grant all permissions to puppetdb]/command: command changed 'notrun' to 'GRANT CREATE ON SCHEMA public TO "puppetdb"'
@@ -61,13 +62,15 @@
puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Puppetdb::Database::Read_only_user[puppetdb-read]/Puppetdb::Database::Default_read_grant[puppetdb grant read permission on new objects from puppetdb to puppetdb-read]/Postgresql_psql[grant default select permission for puppetdb-read]/command: command changed 'notrun' to "ALTER DEFAULT PRIVILEGES\n FOR USER \"puppetdb\"\n IN SCHEMA \"public\"\n GRANT SELECT ON TABLES\n TO \"puppetdb-read\""
puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Puppetdb::Database::Read_only_user[puppetdb-read]/Puppetdb::Database::Default_read_grant[puppetdb grant read permission on new objects from puppetdb to puppetdb-read]/Postgresql_psql[grant default usage permission for puppetdb-read]/command: command changed 'notrun' to "ALTER DEFAULT PRIVILEGES\n FOR USER \"puppetdb\"\n IN SCHEMA \"public\"\n GRANT USAGE ON SEQUENCES\n TO \"puppetdb-read\""
puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Puppetdb::Database::Read_only_user[puppetdb-read]/Puppetdb::Database::Default_read_grant[puppetdb grant read permission on new objects from puppetdb to puppetdb-read]/Postgresql_psql[grant default execute permission for puppetdb-read]/command: command changed 'notrun' to "ALTER DEFAULT PRIVILEGES\n FOR USER \"puppetdb\"\n IN SCHEMA \"public\"\n GRANT EXECUTE ON FUNCTIONS\n TO \"puppetdb-read\""
+ puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Puppetdb::Database::Read_only_user[puppetdb-read]/Puppetdb::Database::Read_grant[puppetdb grant read-only permission on existing objects to puppetdb-read]/Postgresql_psql[grant select permission for puppetdb-read]/command: command changed 'notrun' to "GRANT SELECT\n ON ALL TABLES IN SCHEMA \"public\"\n TO \"puppetdb-read\""
+ puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Puppetdb::Database::Read_only_user[puppetdb-read]/Puppetdb::Database::Read_grant[puppetdb grant read-only permission on existing objects to puppetdb-read]/Postgresql_psql[grant usage permission for puppetdb-read]/command: command changed 'notrun' to "GRANT USAGE\n ON ALL SEQUENCES IN SCHEMA \"public\"\n TO \"puppetdb-read\""
+ puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Puppetdb::Database::Read_only_user[puppetdb-read]/Puppetdb::Database::Read_grant[puppetdb grant read-only permission on existing objects to puppetdb-read]/Postgresql_psql[grant execution permission for puppetdb-read]/command: command changed 'notrun' to "GRANT EXECUTE\n ON ALL FUNCTIONS IN SCHEMA \"public\"\n TO \"puppetdb-read\""
puppet: Notice: /Stage[main]/Puppetdb::Database::Postgresql/Postgresql_psql[grant puppetdb-read role to puppetdb]/command: command changed 'notrun' to 'GRANT "puppetdb-read" TO "puppetdb"'
puppet: Notice: /Stage[main]/Puppetdb::Server/Package[puppetdb]/ensure: created
puppet: Notice: /Stage[main]/Puppetdb::Server::Global/File[/etc/puppetlabs/puppetdb/conf.d/config.ini]/group: group changed 'root' to 'puppetdb'
puppet: Notice: /Stage[main]/Puppetdb::Server::Global/File[/etc/puppetlabs/puppetdb/conf.d/config.ini]/mode: mode changed '0644' to '0640'
- puppet: Notice: /Stage[main]/Puppetdb::Server::Database/File[/etc/puppetlabs/puppetdb/conf.d/database.ini]/owner: owner changed 'root' to 'puppetdb'
puppet: Notice: /Stage[main]/Puppetdb::Server::Database/File[/etc/puppetlabs/puppetdb/conf.d/database.ini]/group: group changed 'root' to 'puppetdb'
- puppet: Notice: /Stage[main]/Puppetdb::Server::Database/File[/etc/puppetlabs/puppetdb/conf.d/database.ini]/mode: mode changed '0644' to '0600'
+ puppet: Notice: /Stage[main]/Puppetdb::Server::Database/File[/etc/puppetlabs/puppetdb/conf.d/database.ini]/mode: mode changed '0644' to '0640'
puppet: Notice: /Stage[main]/Puppetdb::Server::Database/Ini_setting[puppetdb_psdatabase_username]/ensure: created
puppet: Notice: /Stage[main]/Puppetdb::Server::Database/Ini_setting[puppetdb_psdatabase_password]/ensure: created
puppet: Notice: /Stage[main]/Puppetdb::Server::Database/Ini_setting[puppetdb_pgs]/ensure: created
@@ -87,9 +90,8 @@
puppet: Notice: /Stage[main]/Puppetdb::Server::Read_database/Ini_setting[puppetdb_read_subname]/ensure: created
puppet: Notice: /Stage[main]/Puppetdb::Server::Read_database/Ini_setting[puppetdb_read_conn_max_age]/ensure: created
puppet: Notice: /Stage[main]/Puppetdb::Server::Read_database/Ini_setting[puppetdb_read_conn_lifetime]/ensure: created
- puppet: Notice: /Stage[main]/Puppetdb::Server::Jetty/File[/etc/puppetlabs/puppetdb/conf.d/jetty.ini]/owner: owner changed 'root' to 'puppetdb'
puppet: Notice: /Stage[main]/Puppetdb::Server::Jetty/File[/etc/puppetlabs/puppetdb/conf.d/jetty.ini]/group: group changed 'root' to 'puppetdb'
- puppet: Notice: /Stage[main]/Puppetdb::Server::Jetty/File[/etc/puppetlabs/puppetdb/conf.d/jetty.ini]/mode: mode changed '0644' to '0600'
+ puppet: Notice: /Stage[main]/Puppetdb::Server::Jetty/File[/etc/puppetlabs/puppetdb/conf.d/jetty.ini]/mode: mode changed '0644' to '0640'
puppet: Notice: /Stage[main]/Puppetdb::Server::Jetty/File[/etc/puppetlabs/puppetdb/conf.d/jetty.ini]/seluser: seluser changed 'unconfined_u' to 'system_u'
puppet: Notice: /Stage[main]/Puppetdb::Server::Jetty/Ini_setting[puppetdb_host]/ensure: created
puppet: Notice: /Stage[main]/Puppetdb::Server::Jetty/Ini_setting[puppetdb_sslprotocols]/ensure: created
@vchepkov nice, I'd guess the issue is in the changes to the SQL code, I'm assuming the file mode changes are innocuous.
Right, so the command create database fails and that fact doesn't get detected by puppet
postgres=# CREATE DATABASE "puppetdb" WITH TEMPLATE = "template0" ENCODING = 'UTF8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8';
ERROR: invalid locale name: "en_US.UTF-8"
It makes an assumption that locale is installed, but it's not, by default
[postgres@puppet ~]$ locale -a
C
C.utf8
POSIX
So, if module wants to use this locale, it should make sure glibc-langpack-en is installed before postgresql started
https://github.com/puppetlabs/puppetlabs-puppetdb/blob/main/manifests/database/postgresql.pp#L153
Describe the Bug
Upgrading to module v8.1.0 renders puppetdb unoperational , service doesn't start The following log entries are observed in puppetdb.log
Reverting back to v8.0.1 restores functionality
Environment
AlmaLinux 9.4