search

hapostgres / pg_auto_failover

Postgres extension and service for automated failover and high-availability
Other
1.07k stars 112 forks source link

Error when creating Citus worker to a formation #1022

Open machinify-hankhuang opened 6 months ago

machinify-hankhuang commented 6 months ago

I have existing Citus database worker pairs joined pgautofailover formation with pg_autoctl create postgres command using pgautofailover v1.6.3

Now I've upgraded pgautofailover to v2.0. I removed those Citus worker nodes from the formation and tried to re-add them using pg_autoctl create worker command. Everything looks good except it will error out with the following message:

04:44:17 22191 WARN  NOTICE:  extension "pg_stat_statements" already exists, skipping
04:44:17 22191 ERROR Failed to create extension "pg_stat_statements"[XX000]: ERROR:  operation is not allowed on this node
HINT:  Connect to the coordinator and run it again.

It looks like it was going to skip the step of implementing pg_stat_statements extension but it didn't. It feels like a bug. I wasn't able to find a way to move forward with these. But to give more context, here's the full log when I run pg_autoctl create worker:

04:44:16 22187 INFO  Started pg_autoctl postgres service with pid 22190
04:44:16 22190 INFO   /usr/pgsql-13/bin/pg_autoctl do service postgres --pgdata /mnt/lib/pgsql/13/data/ -v
04:44:16 22187 INFO  Started pg_autoctl node-init service with pid 22191
04:44:16 22191 INFO  Registering Postgres system 7276565784674612022 found at "/mnt/lib/pgsql/13/data"
04:44:16 22191 INFO  Registered node 37 "worker1a" (test-c-worker-0-p:5432) in formation "test-c-worker-0", group 1, state "single"
04:44:16 22191 INFO  Writing keeper state file at "/var/lib/pgsql/.local/share/pg_autoctl/mnt/lib/pgsql/13/data/pg_autoctl.state"
04:44:16 22191 INFO  Writing keeper init state file at "/var/lib/pgsql/.local/share/pg_autoctl/mnt/lib/pgsql/13/data/pg_autoctl.init"
04:44:16 22191 INFO  Successfully registered as "single" to the monitor.
04:44:16 22191 INFO  FSM transition from "init" to "single": Start as a single node
04:44:16 22191 INFO  Initialising postgres as a primary
04:44:16 22191 WARN  Skipping HBA edits (per --skip-pg-hba) for rule: hostssl "claim" "postgres" test-c-worker-0-p trust
04:44:16 22191 INFO  Contents of "/mnt/lib/pgsql/13/data/postgresql-auto-failover.conf" have changed, overwriting
04:44:16 22197 INFO   /usr/pgsql-13/bin/postgres -D /mnt/lib/pgsql/13/data -p 5432 -h *
04:44:17 22191 INFO  The user "postgres" already exists, skipping.
04:44:17 22191 INFO  CREATE DATABASE citus;
04:44:17 22191 INFO  The database "citus" already exists, skipping.
04:44:17 22191 INFO  CREATE EXTENSION pg_stat_statements;
04:44:17 22191 WARN  NOTICE:  extension "pg_stat_statements" already exists, skipping
04:44:17 22191 ERROR Failed to create extension "pg_stat_statements"[XX000]: ERROR:  operation is not allowed on this node
HINT:  Connect to the coordinator and run it again.

04:44:17 22191 ERROR Failed to create extension pg_stat_statements
04:44:17 22191 ERROR Failed to transition from state "init" to state "single", see above.
04:44:17 22187 ERROR pg_autoctl service node-init exited with exit status 12

Thanks,

cc. @DimCitus

Pookping commented 4 months ago

Hi. @machinify-hankhuang were you ever able to resolve this problem? I have the same problem. Thank you.

machinify-hankhuang commented 3 months ago

@Pookping No, I wasn't able to and I believe this is a bug.