RSA_PSS_PSS_SHA256 not working with tls1.3

[Nellta]

Hello!

I started a server with the following command in the apps directory:

sudo java -jar TLS-Server.jar -signature_hash_algo RSA_PSS_PSS_SHA256 -port 4433  -config ../resources/configs/tls13

And then I simply try to connect to it with openssl:

sudo openssl s_client -connect localhost:4433 -sigalgs rsa_pss_pss_sha256

However this fails and I get the following output:

14:32:21 [main] INFO : ServerTcpTransportHandler - Connection established from ports 4433 -> 54140
14:32:21 [main] INFO : TlsAction - Received Messages (server): CLIENT_HELLO,
14:32:21 [main] INFO : SendAction - Sending messages (server): SERVER_HELLO, CHANGE_CIPHER_SPEC, ENCRYPTED_EXTENSIONS, CERTIFICATE, CERTIFICATE_VERIFY, FINISHED,
14:32:22 [main] WARN : RecordDecryptor - Failed to decrypt record, will try to process with previous cipher
14:32:22 [main] INFO : TlsAction - Received Messages (server): Alert(FATAL,ILLEGAL_PARAMETER),
14:32:22 [main] INFO : DefaultWorkflowExecutor - Workflow was not executed as planned.

I am wondering if I'm missing some line in the default tls 1.3 config file I am using?

PS. I forgot to add that I have edited the default tls13.config file to support RSA_PSS_PSS. instead of RSA_PSS_RSAE.

UPDATE:

I later tried to simply supply my own self signed certificate for rsa_pss_pss_sha384 (I switched to sha384) which I created with:

sudo openssl req -new -x509 -newkey RSA-PSS -keyout rsa_pss_pss_sha384.key  -sigopt rsa_padding_mode:pss -sha384  -out rsa_pss_pss_sha384.cert -nodes

And when I then ran the same command with the -cert /-key parameters added I instead got a new warning:

11:36:02 [main] WARN : CertificateKeyPair - Unknown algorithm ID: 1.2.840.113549.1.1.10 using "NONE"
11:36:02 [main] WARN : CertificateKeyPair - Unknown algorithm ID: 1.2.840.113549.1.1.10 using "ANONYMOUS_NONE"

Which turned out to be because the above ID for rsa_psspss* is not implemented in the CertificateKeyPair.java file. If I edited that file to return CertificateKeyType.RSA and SignatureAndHashAlgorithm.RSA_PSS_PSS_SHA384 as the default case and then rebuild the whole project, the connection is established with the correct signature.

Am I correct in assuming that rsa_pss_pss aren't fully implemented, or did I just do something weird that turned out to work?

As a final note the above self signed certificate works fine if I run it with openssl s_server and try to make a simple connection with openssl s_client

[ic0ns]

Yes I think PSS_PSS is not yet fully implemented. We have a prototype internally which will 'eventually' reach the public master. However, it is part of a bigger rework and will probably take some time...

[Nellta]

Ok, thanks for the answer. I'll just have to wait for the next release!