empty md5 from serialized der / lengthy md5 from empty string

[aronatkins]

We are using openssl::md5 to create a short "token" from an RSA public key.

The relevant code is:

  key <- openssl::rsa_keygen(2048L)
  priv.der <- openssl::write_der(key)
  pub.der <- openssl::write_der(key$pubkey)
  tokenId <- as.character(openssl::md5(serialize(pub.der, NULL)))

Full code: https://github.com/rstudio/rsconnect/blob/5e74f65b23d01b45295b8ce2e50c8cdef9e25abc/R/rsa.R#L5-L21

We have a user who is seeing that the result from openssl::md5 is an empty string. We have asked them to send the output from each of the sub-steps and this is what we've got:

> key <- openssl::rsa_keygen(2048L)
> print(key)
[2048-bit rsa private key]
md5:

> priv.der <- openssl::write_der(key)
> print(head(priv.der))
[1] 30 82 04 a4 02 01

> pub.der <- openssl::write_der(key$pubkey)
> print(head(pub.der))
[1] 30 82 01 22 30 0d

> serialized <- serialize(pub.der, NULL)
> print(head(serialized))
[1] 58 0a 00 00 00 03

> tokenId <- as.character(openssl::md5(serialized))
> print(tokenId)
[1] ""

print(openssl::md5(""))
[1] "d0cdeb8bab5......."  # produces 43862 characters

We do not see an error from openssl::md5 but the output is unexpected in both cases.

The R version running is 3.6.0 And the openssl version is 1.4.1 (from CRAN)

Have you seen this before? Any troubleshooting/debugging advice?

[jeroen]

No I have never seen this. Can you ask to include sessionInfo() and openssl::openssl_config()

[jmcphers]

I have seen this kind of behavior in OpenSSL on systems that have FIPS compliance. FIPS generally disallows MD5 since it isn't considered a secure algorithm, and OpenSSL can be made to respect this decision.

[aronatkins]

Here is a Dockerfile that shows openssl::md5 producing an empty string in FIPS mode.

FROM centos:centos7

RUN yum -y install epel-release

RUN yum -y install \
        dracut-fips \
        openssl-devel \
        R-core \
        R-core-devel

RUN R --slave --vanilla -e 'install.packages("openssl", repos = "https://cran.rstudio.com/")'

ENV OPENSSL_FORCE_FIPS_MODE=1
CMD ["R", "--slave", "--vanilla", "-e", "openssl::md5('')"]

Build it:

docker build -t openssl-empty-md5 .

Run it:

docker run -it openssl-empty-md5
# => [1] ""

[aronatkins]

The R_md_init function may need to check the response from EVP_DigestInit_ex: https://github.com/jeroen/openssl/blob/b78f7d505383447312ca3d0df007cb107ae0f90f/src/stream.c#L21

That function returns 1 on success: https://www.openssl.org/docs/man1.1.0/man3/EVP_get_digestbyname.html

Calls to EVP_DigestUpdate and EVP_DigestFinal_ex may also need error handling, but the initialization call is probably enough to make the openssl::md5 call err when the algo is not supported.

It looks like lots of other projects (Python, Ruby, etc) have all encountered this issue in different ways. Here's a Ruby issue where unchecked initialization led to a crash: https://redmine.ruby-lang.org/issues/4944

[aronatkins]

We have changed rsconnect to not useopenssl::md5 when creating tokens. (https://github.com/rstudio/rsconnect/issues/378)

[jeroen]

@aronatkins thanks for the reproducible example! I was able to reproduce it in centos 7 by setting the OPENSSL_FORCE_FIPS_MODE variable as you show.

I have now added a check that should catch this problem and error, so now you should see:

> library(openssl)
> md5("foo")
Error: OpenSSL error in EVP_DigestInit_ex: disabled for fips

Of course that doesn't help you but it's still useful :)