Invalid size in RSA/sigGen SHAKE-128 in PSS/mgf1

[sandor-szendro-i4p]

environment Demo

testSessionId 428747

vsId 1774510

Algorithm registration { "revision":"FIPS186-5", "algorithm":"RSA", "mode":"sigGen", "capabilities":[ { "sigType":"pkcs1v1.5", "properties":[ { "modulo":2048, "hashPair":[ { "hashAlg":"SHA2-224" }, { "hashAlg":"SHA2-256" }, { "hashAlg":"SHA2-384" }, { "hashAlg":"SHA2-512" }, { "hashAlg":"SHA3-224" }, { "hashAlg":"SHA3-256" }, { "hashAlg":"SHA3-384" }, { "hashAlg":"SHA3-512" } ] }, { "modulo":3072, "hashPair":[ { "hashAlg":"SHA2-224" }, { "hashAlg":"SHA2-256" }, { "hashAlg":"SHA2-384" }, { "hashAlg":"SHA2-512" }, { "hashAlg":"SHA3-224" }, { "hashAlg":"SHA3-256" }, { "hashAlg":"SHA3-384" }, { "hashAlg":"SHA3-512" } ] }, { "modulo":4096, "hashPair":[ { "hashAlg":"SHA2-224" }, { "hashAlg":"SHA2-256" }, { "hashAlg":"SHA2-384" }, { "hashAlg":"SHA2-512" }, { "hashAlg":"SHA3-224" }, { "hashAlg":"SHA3-256" }, { "hashAlg":"SHA3-384" }, { "hashAlg":"SHA3-512" } ] } ] }, { "sigType":"pss", "properties":[ { "modulo":2048, "maskFunction":[ "mgf1" ], "hashPair":[ { "hashAlg":"SHA2-224", "saltLen":28 }, { "hashAlg":"SHA2-256", "saltLen":32 }, { "hashAlg":"SHA2-384", "saltLen":48 }, { "hashAlg":"SHA2-512", "saltLen":64 }, { "hashAlg":"SHA3-224", "saltLen":28 }, { "hashAlg":"SHA3-256", "saltLen":32 }, { "hashAlg":"SHA3-384", "saltLen":48 }, { "hashAlg":"SHA3-512", "saltLen":64 }, { "hashAlg":"SHAKE-128", "saltLen":16 }, { "hashAlg":"SHAKE-256", "saltLen":32 } ] }, { "modulo":3072, "maskFunction":[ "mgf1" ], "hashPair":[ { "hashAlg":"SHA2-224", "saltLen":28 }, { "hashAlg":"SHA2-256", "saltLen":32 }, { "hashAlg":"SHA2-384", "saltLen":48 }, { "hashAlg":"SHA2-512", "saltLen":64 }, { "hashAlg":"SHA3-224", "saltLen":28 }, { "hashAlg":"SHA3-256", "saltLen":32 }, { "hashAlg":"SHA3-384", "saltLen":48 }, { "hashAlg":"SHA3-512", "saltLen":64 }, { "hashAlg":"SHAKE-128", "saltLen":16 }, { "hashAlg":"SHAKE-256", "saltLen":32 } ] }, { "modulo":4096, "maskFunction":[ "mgf1" ], "hashPair":[ { "hashAlg":"SHA2-224", "saltLen":28 }, { "hashAlg":"SHA2-256", "saltLen":32 }, { "hashAlg":"SHA2-384", "saltLen":48 }, { "hashAlg":"SHA2-512", "saltLen":64 }, { "hashAlg":"SHA3-224", "saltLen":28 }, { "hashAlg":"SHA3-256", "saltLen":32 }, { "hashAlg":"SHA3-384", "saltLen":48 }, { "hashAlg":"SHA3-512", "saltLen":64 }, { "hashAlg":"SHAKE-128", "saltLen":16 }, { "hashAlg":"SHAKE-256", "saltLen":32 } ] } ] }

Endpoint in which the error is experienced https://demo.acvts.nist.gov/acvp/v1/testSessions GET

Expected behavior For RSA/sigGen when sigType is "pss" and maskFunction is "mgf1" and hashAlg is SHAKE-128 the expected results returned by ACVP server only the first 16 bytes of the 32 bytes of the SHAKE-128 output is used in the mask generation function.

Additional context According to FIPS 186-5 5.4.1 Mask Generation Functions in RSASSA-PSS refers to B.2.1 of RFC 8017. B.2.1 of RFC 8017 contains the steps for using the mask generation function, where step 3 is: "For counter from 0 to \ceil (maskLen / hLen) - 1, do the following:" By examination of the expected test vectors we think in this step instead of hLen, hLen / 2 is used. This means for SHAKE-128 instead of maskLen / 32 -1, maskLen / 16 -1 is used. We think maskLen / 32 -1 should be used.

In case of SHAKE-256 32 bytes are used instead of 64.

[jbrock24]

@sandor-szendro-i4p Looking into this

[livebe01]

Thanks for letting us know. I think I see where the issue is. I'm working on it now. I'll give you an update when it's been fixed.

[livebe01]

We have a fix in for this. It will go out in the next release.

[szendros]

Thanks for the fix, I will be able to test it on monday.

[livebe01]

Thank you!

[livebe01]

The fix for this is on Demo in release v1.1.0.31.

[szendros]

I tested it on Demo and it works for me. Thank you, this can be closed.

[livebe01]

Thanks. Appreciate the confirmation!

[livebe01]

The fix for this is on Prod in release v1.1.0.31.