[Bug]: The microservice type import doesnt work as expected

radhupr commented 1 month ago

Is there an existing issue already for this bug?

[x] I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

[x] I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

[x] I have read the troubleshooting guide and I think this is a new bug.

Contact Details

No response

Version

1.25 (latest patch)

What version of Kubernetes are you using?

1.32

What is your Kubernetes environment?

Self-managed: k0s

How did you install the operator?

Helm

What happened?

My source DB is of version 16.4 and I'm testing main version upgrade using the import method

psql (16.4 (Debian 16.4-1.pgdg110+2))
Type "help" for help.

postgres=# \l
                                                        List of databases
   Name    |  Owner   | Encoding | Locale Provider | Collate | Ctype | ICU Locale | ICU Rules |        Access privileges
-----------+----------+----------+-----------------+---------+-------+------------+-----------+----------------------------------
 postgres  | postgres | UTF8     | libc            | C       | C     |            |           | =Tc/postgres                    +
           |          |          |                 |         |       |            |           | postgres=CTc/postgres           +
           |          |          |                 |         |       |            |           | cnpg_pooler_pgbouncer=c/postgres
 template0 | postgres | UTF8     | libc            | C       | C     |            |           | =c/postgres                     +
           |          |          |                 |         |       |            |           | postgres=CTc/postgres
 template1 | postgres | UTF8     | libc            | C       | C     |            |           | =c/postgres                     +
           |          |          |                 |         |       |            |           | postgres=CTc/postgres
 testapp   | testapp  | UTF8     | libc            | C       | C     |            |           |
(4 rows)

I used below yaml for creating a DB to import database testapp

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: testdb-17
  namespace: database
spec:
  instances: 2
  resources:
    limits:
      ephemeral-storage: 10Gi
      memory: 8Gi
    requests:
      cpu: 2000m
      ephemeral-storage: 3Gi
      memory: 3Gi
  storage:
    size: 2Gi
    storageClass: ontap-nas
  bootstrap:
    initdb:
      import:
        type: microservice
        schemaOnly: true
        databases:
          - testapp
        source:
          externalCluster: testdb
  externalClusters:
  - name: testdb
    connectionParameters:
      host: testdb-rw.database.svc
      user: postgres
      dbname: postgres
    password:
      name: testdb-superuser
      key: password

When the new cluster is created, its missing testapp but have app DB

postgres=# \l
                                                List of databases
   Name    |  Owner   | Encoding | Locale Provider | Collate | Ctype | Locale | ICU Rules |   Access privileges
-----------+----------+----------+-----------------+---------+-------+--------+-----------+-----------------------
 app       | app      | UTF8     | libc            | C       | C     |        |           |
 postgres  | postgres | UTF8     | libc            | C       | C     |        |           |
 template0 | postgres | UTF8     | libc            | C       | C     |        |           | =c/postgres          +
           |          |          |                 |         |       |        |           | postgres=CTc/postgres
 template1 | postgres | UTF8     | libc            | C       | C     |        |           | =c/postgres          +
           |          |          |                 |         |       |        |           | postgres=CTc/postgres
(4 rows)

I tried to change it to monolith type as below, but that failed with another error: Error: while executing logical import: error while executing pg_restore, section:pre-data, exit status 1

bootstrap:
    initdb:
      import:
        schemaOnly: true
        type: monolith
        roles:
         - testapp
        databases:
          - testapp
        source:
          externalCluster: testdb
  externalClusters:
  - name: testdb
    connectionParameters:
      host: testdb-rw.database.svc
      user: postgres
      dbname: postgres
    password:
      name: testdb-superuser
      key: password

Can anyone help in understanding if there is any issue in how I import the initdb from source?

Cluster resource

#source cluster yaml

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  labels:
    app.kubernetes.io/instance: cluster
    app.kubernetes.io/name: testdb
  name: testdb
  namespace: database
spec:
  affinity:
    topologyKey: topology.kubernetes.io/zone
  bootstrap:
    initdb:
      database: testapp
      encoding: UTF8
      owner: testapp
      secret:
        name: "db-creds"
  enablePDB: true
  enableSuperuserAccess: true
  #imageName: ghcr.io/cloudnative-pg/postgresql:17.4
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
  instances: 2
  logLevel: warning
  monitoring:
    disableDefaultQueries: false
    enablePodMonitor: true
  ...
  ...

Relevant log output

Code of Conduct

[x] I agree to follow this project's Code of Conduct

dosubot[bot] commented 1 month ago

Hello @radhupr! I'm Dosu, your friendly bot here to help you with bugs, answer your questions, and guide you as you contribute. I'm here while you wait for a human maintainer. Let's tackle this issue together!

The issue you're experiencing with the microservice type import not working as expected might be related to the database name mapping during the import process. In your configuration, the database name specified for import is testapp, but the resulting database in the new cluster is named app. This discrepancy could be due to incorrect specification or mapping of the database name in the import configuration.

Here are a few things to check and consider:

Database Name Mapping: Ensure that the databases field in your import configuration correctly matches the intended database name in the source cluster. The dbname parameter in the connectionParameters should also match the database name you want to import ^[1].
Schema-Only Import: When using the schemaOnly option, ensure that the database schema is correctly defined and matches the source database. This option only imports the schema without data, so verify that the schema exists and is correctly specified.
Monolith Type Error: The error error while executing pg_restore, section:pre-data, exit status 1 indicates a failure during the pg_restore process. This could be due to missing dependencies or incorrect configurations in the pre-data section. Ensure that all necessary roles and permissions are correctly set up in the source cluster and that the roles field in your configuration matches the roles in the source cluster.
PostgreSQL Version Compatibility: Ensure that the PostgreSQL version of the destination cluster is compatible with the source cluster. The destination cluster's version should be greater than or equal to the source cluster's version to support major upgrades ^[2].
Connection Parameters: Double-check the connection parameters, including host, user, and password, to ensure they are correctly configured and can access the source cluster.

By addressing these points, you should be able to resolve the discrepancies and errors during the import process.

To continue talking to Dosu, mention @dosu.

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

radhupr commented 1 month ago

mentioning initdb resolved this issue.

 bootstrap:
    initdb:
      database: testapp
      encoding: UTF8
      # owner: testapp
      import:
        type: microservice
        schemaOnly: true
        databases:
          - testapp
        source:
          externalCluster: testdb

I was trying to use monolith option but the DB creation fails "level":"info","ts":"2025-05-13T12:37:07.52692411Z","msg":"Exited log pipe","fileName":"/controller/log/postgres.csv","logging_pod":"testdb-upg-monolith-1-import"} {"level":"error","ts":"2025-05-13T12:37:07.526910795Z","msg":"Error while bootstrapping data directory","logging_pod":"testdb-upg-monolith-1-import","error":"while executing logical import: error while executing pg_restore, section:pre-data, exit status 1","stacktrace":"github.com/cloudnative-pg/machinery/pkg/log.(*logger).Error\n\tpkg/mod/github.com/cloudnative-pg/machinery@v0.1.0/pkg/log/log.go:125\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/initdb.initSubCommand\n\tinternal/cmd/manager/instance/initdb/cmd.go:160\ngithub.com/cloudnative-pg/cloudnative-pg/internal/cmd/manager/instance/initdb.NewCmd.func2\n\tinternal/cmd/manager/instance/initdb/cmd.go:109\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1015\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v1.9.1/command.go:1071\nmain.main\n\tcmd/manager/main.go:68\nruntime.main\n\t/opt/hostedtoolcache/go/1.24.0/x64/src/runtime/proc.go:283"} Error: while executing logical import: error while executing pg_restore, section:pre-data, exit status 1

spec is as follows

bootstrap:
    initdb:
      import:
        type: monolith
        roles:
          - testapp
          - upgrade
        databases:
          - testapp
          - upgrade
        source:
          externalCluster: testdb
  externalClusters:
  - name: testdb
    connectionParameters:
      host: testdb-rw.database.svc
      user: postgres
      dbname: postgres
      sslmode: require
    password:
      name: testdb-superuser
      key: password

cloudnative-pg / cloudnative-pg