failed to create a new keydbstore: pq: unknown authentication response: 10

[kevinlmadison]

I'm using the Harbor Operator to deploy a harbor cluster and in my notary server and signer pods I'm getting the following output

2021/10/05 16:55:06 Updating database.
2021/10/05 16:55:06 schema_migrations table does not require update, skip.
no change
notarysigner database migrated to latest version
{"level":"info","msg":"Version: 0.6.1, Git commit: d6e1431f","time":"2021-10-05T16:55:06Z"}
{"level":"fatal","msg":"failed to create a new keydbstore: pq: unknown authentication response: 10","time":"2021-10-05T16:55:06Z"}

I believe this has to do with the scram-sha-256 client library needing to be updated as per this post on stackoverflow.

Is this something I can work around or does this need to be fixed in the codebase? Thank you!

[wy65701436]

hi, could you please file issue on harbor operator repo?

[zer0def]

This issue isn't operator-specific, but rather specifically applicable to both notary services. This still applies in 2.4.0.

[InderpreetSaini]

Hi, Did you find any resolution to this issue? I am facing the same issue for both notary server and signer pods? I am using the below postgres cluster.

https://access.crunchydata.com/documentation/postgres-operator/5.0.4/quickstart/

Harbor was deployed using Helm charts for Harbor HA setup.

[kevinlmadison]

Hey @InderpreetSaini I am using the same operator setup for postgres, and essentially what you have to do is update the verifier in the secret created by crunchy with you postgres database credentials. The update you need to make is to hash with MD5 instead of scramsha256. There is some info here There was another page where they showed you how to hash and I'm looking for that for you now.

[kevinlmadison]

@InderpreetSaini Here you go! He links to the code you can use in his response to me.

[InderpreetSaini]

Thanks a lot @kevinlmadison ! It worked and now my notary pods are running without any issues.

[djtaylor]

I was having the same issue, but the link mentioned above worked for me in resolving it. Are there any plans on adding support for SCRAM-SHA-256?

[Ramki33]

@InderpreetSaini @kevinlmadison

HI ,
i am facing same issue , with harbor using postgres operator I am using the postgres cluster https://access.crunchydata.com/documentation/postgres-operator/5.0.4/quickstart/

i tried patching secret with SCRAM-SHA-256 as mentioned above its not solving :-( could you pls guide me

oc logs harbor-notary-signer-5d5bf66dbb-p9kzb 2022/04/13 20:25:12 Updating database. 2022/04/13 20:25:12 schema_migrations table does not require update, skip. no change notarysigner database migrated to latest version {"level":"info","msg":"Version: 0.6.1, Git commit: d6e1431f","time":"2022-04-13T20:25:12Z"} {"level":"fatal","msg":"failed to create a new keydbstore: pq: unknown authentication response: 10","time":"2022-04-13T20:25:12Z"}

harbor values.yaml : external: host: "hippo-primary.postgres-operator.svc" port: "5432" username: hippo password: datalake coreDatabase: "registry" notaryServerDatabase: "notaryserver" notarySignerDatabase: "notarysigner" sslmode: "require"

[Ramki33]

I was having the same issue, but the link mentioned above worked for me in resolving it. Are there any plans on adding support for SCRAM-SHA-256?

hi @djtaylor ,

I tried above mentioned step . its not working for me ( see my comment above ) i might be doing some thing wrong . could you guide me.

[InderpreetSaini]

@InderpreetSaini @kevinlmadison

HI , i am facing same issue , with harbor using postgres operator I am using the postgres cluster https://access.crunchydata.com/documentation/postgres-operator/5.0.4/quickstart/

i tried patching secret with SCRAM-SHA-256 as mentioned above its not solving :-( could you pls guide me

oc logs harbor-notary-signer-5d5bf66dbb-p9kzb 2022/04/13 20:25:12 Updating database. 2022/04/13 20:25:12 schema_migrations table does not require update, skip. no change notarysigner database migrated to latest version {"level":"info","msg":"Version: 0.6.1, Git commit: d6e1431f","time":"2022-04-13T20:25:12Z"} {"level":"fatal","msg":"failed to create a new keydbstore: pq: unknown authentication response: 10","time":"2022-04-13T20:25:12Z"}

harbor values.yaml : external: host: "hippo-primary.postgres-operator.svc" port: "5432" username: hippo password: datalake coreDatabase: "registry" notaryServerDatabase: "notaryserver" notarySignerDatabase: "notarysigner" sslmode: "require"

Hi @Ramki33 , as per my understanding, notary in Harbor does not support SCRAM-SHA-256 encrypted password authentication. Have you tried generating the verifier using algorithm value as md5 instead of scram-sha-256?

Besides the latest version of Harbor 2.5.0 supports Cosign integration as an image signing solution along with Notary. So you have the option to go along with Cosign, instead of Notary.

[Ramki33]

Hi @InderpreetSaini ,

Thanks a lot it worked with md5 as you suggested .

[bbroniewski]

Just use some tool to hash your password with md5 (postgres uses also username to hash) and when you do it, then alter the user like:

ALTER USER test WITH PASSWORD 'md5c4c2d8de69efb9041d37711fe303a3dd'

Then password will be stored as MD5 encrypted and all pods will work.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

[mubei1005]

Resolution to this issue, you need to make postgres auth to hash with MD5 instead of scramsha256. Maybe you used PG client lib not support scram-sha-256. fixed: step 1: configue password_encryption with md5 in postgresql.conf

root@postgresql:/# cat /var/lib/postgresql/data/postgresql.conf | grep password_encryption
#password_encryption = scram-sha-256    # scram-sha-256 or md5
password_encryption = md5

step 2: configue md5 with METHOD in pg_hba.conf

root@postgresql-9b4f84c8d-lsdhs:/# cat /var/lib/postgresql/data/pg_hba.conf  | grep md5
host    all             all             0.0.0.0/0        md5

step 3: reload your pstgresql

pg_ctl reload -D $PGDATA  

Make sure postgresql use md5 storage password.

password_encryption
---------------------
 md5
(1 row)

step 4: change your db user password;

ALTER USER user03 WITH PASSWORD 'your_password';

postgres=# select rolname,rolpassword from pg_authid;
          rolname          |       rolpassword    
---------------------------+-------------------------------
 pg_checkpoint             | 
 root                      | 
 postgres                  | SCRAM-SHA-256$4096:Iob05p7...
 user01                    | SCRAM-SHA-256$4096:S4sa7p7...
 user02                    | SCRAM-SHA-256$4096:Iob05ix...
 user03                    | md58e4e6cccbebf93f36c08549...

Now user03's password pre md5

Last, restart your notaryserver adn notarysigner. It's working.

[Smartich0ke]

Just adding to @mubei1005 's comment, don't forget to make sure your pg_hba.conf is updated accordingly otherwise you might not be able to login.

For example, to make sure the user 'harbor' has their password hashed with md5, add the following line (above any host all all all ... lines):

host    all     harbor    all     md5