libtls: Signature algorithm extension fails for openssl

[bytinbit]

Aufwandschätzung: soll=2 Stunden

After we implemented the signature algorithm extension, the connection succeeds with the test server/client within strongswan. However, if we do the same using the tls-client from strongswan and try to connect to an openssl tls 1.3 server, the connection fails with the following error on the server side:

Using default temp DH parameters
ACCEPT
read from 0x55f99b8071f0 [0x55f99b814473] (5 bytes => 5 (0x5))
0000 - 16 03 03 00 bd                                    .....
read from 0x55f99b8071f0 [0x55f99b814478] (189 bytes => 189 (0xBD))
0000 - 01 00 00 b9 03 03 5e 7f-54 34 5c 21 ce e2 05 9f   ......^.T4\!....
0010 - 35 df 5c 6b 10 7a eb d9-3c 44 3d 92 9a a9 c0 1f   5.\k.z..<D=.....
0020 - ab 7c e0 ef 68 44 00 00-2c 13 01 13 02 13 03 13   .|..hD..,.......
0030 - 04 13 05 00 33 00 67 00-39 00 6b 00 9e 00 9f 00   ....3.g.9.k.....
0040 - 16 00 2f 00 3c 00 35 00-3d 00 9c 00 9d 00 0a 00   ../.<.5.=.......
0050 - 02 00 3b 00 01 01 00 00-64 00 00 00 13 00 11 00   ..;.....d.......
0060 - 00 0e 77 77 77 2e 74 65-73 74 2e 6c 6f 63 61 6c   ..www.test.local
0070 - 00 0a 00 04 00 02 00 1d-00 2b 00 09 08 03 04 03   .........+......
0080 - 03 03 02 03 01 00 0d 00-0a 00 08 04 01 05 01 06   ................
0090 - 01 02 01 00 33 00 26 00-24 00 1d 00 20 19 1c cd   ....3.&.$... ...
00a0 - 43 11 27 6f 94 78 e9 20-79 6c c4 0f 73 f2 9e 24   C.'o.x. yl..s..$
00b0 - e4 c5 b5 99 b0 71 81 da-99 92 1a 5e 26            .....q.....^&
write to 0x55f99b8071f0 [0x55f99b81d690] (7 bytes => 7 (0x7))
0000 - 15 03 03 00 02 02 28                              ......(
ERROR
140038109491648:error:14201076:SSL routines:tls_choose_sigalg:no suitable signature algorithm:../ssl/t1_lib.c:2589:
shutting down SSL
CONNECTION CLOSED

We sent an RSA with SHA256, but openssl doesn't recognise it as suitable signature algorithm. However, if we do the same with a strongswan client max version of TLS 1.2, the connection succeeds, using RSA with SHA256.