Not able to Enroll with NDES, failure reason: Integrity check failed, illegal size of payload

[kothabhaskar]

Hi Everyone,

I am facing an issue when doing a Enroll request with NDES server, We are getting 200 status code and payload but while doing the pkcs_unwrap we are seeing issue, which is listed below for reference. Request you to help us to resolve this issue. Thanks in advance.

Logs for reference:

./sscep: server returned status code 200 ./sscep: MIME header: x-pki-message ./sscep: valid response from server ./sscep: reading outer PKCS#7 ./sscep: PKCS#7 payload size: 713 bytes ./sscep: printing PEM fomatted PKCS#7 ./sscep: PKCS#7 contains 1 bytes of enveloped data ./sscep: verifying signature ./sscep: signature ok ./sscep: finding signed attributes ./sscep: finding attribute transId ./sscep: allocating 32 bytes for attribute ./sscep: reply transaction id: 3144EDBD9AA4A9D9FFD8C82693C4C2E8 ./sscep: finding attribute messageType ./sscep: allocating 1 bytes for attribute ./sscep: reply message type is good ./sscep: finding attribute senderNonce ./sscep: allocating 16 bytes for attribute ./sscep: senderNonce in reply: 46A5BA0F22DD3C4BBB1781EB22119225 ./sscep: finding attribute recipientNonce ./sscep: allocating 16 bytes for attribute ./sscep: recipientNonce in reply: 8E2F075218421ADDC6D1E2668CAD0C7E ./sscep: finding attribute pkiStatus ./sscep: allocating 1 bytes for attribute ./sscep: pkistatus: FAILURE ./sscep: finding attribute failInfo ./sscep: allocating 1 bytes for attribute ./sscep: reason: Integrity check failed ./sscep: illegal size of payload

[seho85]

Hey,

i also stumbled about hat problem. I'm also using Windows NDES (on Server 2012R2)

It cost me a few hours to figure out the problem:

1) Change in mkrequest script (at line 114) ... [ req ] string_mask = nombstr ...

2) I created the request with password ./mkrequest -ip x.x.x.x 092719FD707E5B0F

3) The NDES Service uses two different certificates one for signing and the other one for encrypting (at least it does so on Server2012R2) They were downloaded when calling sscep getca as ca.crt-0 Certificate for signing ca.crt-1 Certificate for encryption ca.crt-2 Certficate of CA that signed the NDES-Certificates

4) Using this certificates I was able to enroll the certificate ./sscep enroll -c ca.crt-0 -k local.key -r local.csr -l local.crt -e ca.crt-1 -u http://[Server]/certsrv/mscep/mscep.dll

[mbartosch]

Thanks for the analysis. This is very interesting, other SCEP servers we tested (e. g. OpenXPKI, Nexus CM) always return the SCEP RA (encryption) certificate as first certificate. Looks like a special Microsoft-ism. I am afraid a client side heuristic is necessary to determine the correct certificate, then.

[seho85]

I found some time to have a more detailed look to the problem.

I saw that the Certificates returned by the getca command, contain different x509v3 extensions (at least they do on the Server2012R2 NDES Service)

For ca-crt.0: ... X509v3 extensions: 1.3.6.1.4.1.311.20.2: .,.E.n.r.o.l.l.m.e.n.t.A.g.e.n.t.O.f.f.l.i.n.e` ...

For ca-crt.1: ... X509v3 extensions: 1.3.6.1.4.1.311.20.2: ...C.E.P.E.n.c.r.y.p.t.i.o.n ...

I think that could be used for determining the encryption and signing certificate.

Currently im very busy with other stuff, when I found some time I gonna have a look how to integrate that.

Kind regards, Sebastian

[gotthardp]

I discovered another server that does not return SCEP RA (encryption) certificate as the first certificate. To cope with that I made some basic "heuristic" to select the right certficate of those returned from getca. If anyone ever comes across that, the initial logic is in the develop branch, commit 6489e20578355681c5e84c23df5d8104ba6e9df3.

[gotthardp]

Similar to #39, likely solved in 0.8.1. If the problem persists, please open a new issue.