Implement alternative to OpenSSL RSA keygen

[aronatkins]

A customer has an issue with this code: https://github.com/rstudio/rsconnect/blob/1306eed851a01506a81a3f80bf7a92106f7e5a8b/R/rsa.R#L39-L42

The call to openssl::rsa_keygen is producing a FIPS error:

> rsconnect::connectUser(server='mylocaldeployserver')
Error: OpenSSL error in EVP_DigestInit_ex: disabled for fips

This issue is closely related to #378, which hit FIPS errors while using MD5.

(Internal: See support ticket 51376).

[aronatkins]

Here is a Dockerfile showing the error coming from openssl::rsa_keygen (called by rsconnect:::generateToken):

FROM centos:centos7

RUN yum -y install epel-release

RUN yum -y install \
        dracut-fips \
        libcurl-devel \
        openssl-devel \
        wget

# Install R
ARG R_VERSION=3.6.1
ARG OS_IDENTIFIER=centos-7
RUN wget https://cdn.rstudio.com/r/${OS_IDENTIFIER}/pkgs/R-${R_VERSION}-1-1.x86_64.rpm && \
    yum -y install ./R-${R_VERSION}-1-1.x86_64.rpm && \
    ln -s /opt/R/${R_VERSION}/bin/R /usr/bin/R && \
    ln -s /opt/R/${R_VERSION}/bin/Rscript /usr/bin/Rscript && \
    ln -s /opt/R/${R_VERSION}/lib/R /usr/lib/R && \
    rm R-${R_VERSION}-1-1.x86_64.rpm && \
    yum clean all

RUN R --slave --vanilla -e 'install.packages(c("openssl"), repos = "https://cran.rstudio.com/")'

ENV OPENSSL_FORCE_FIPS_MODE=1
CMD ["Rscript", "-e", "openssl::rsa_keygen(2048L)"]

docker build -t rsconnect-fips -f Dockerfile.openssl_rsa .
docker run --rm rsconnect-fips
# => Error: OpenSSL error in EVP_DigestInit_ex: disabled for fips
# => Execution halted

[ricardofandrade]

It may be key length: https://github.com/openssl/openssl/issues/11863 https://access.redhat.com/discussions/1518473

Or key format: https://niranjanmr.wordpress.com/2011/11/15/rhel6-fips/

[ricardofandrade]

A workaround is using API keys:

> rsconnect::addConnectServer("https://example.com", "example")
Server 'example' added successfully: https://example/__api__
> rsconnect::connectApiUser("user", "example", "<api key>")
Account registered successfully: user

[JosiahParry]

In the python library hashlib the .md5 function has argument usedForSecurity which, when set to False enables the use of md5 when fips is enables. This is the motivations behind https://docs.rstudio.com/connect/news/#rstudio-connect-1881. Is it possible to implement the same thing with rsconnect? That would permit my customers to connect via the gui

[JosiahParry]

This issue has been resolved for python users on RSC. However this issue continues to persist for R users. Is there a plan to resolve this? The use of FIPS is a challenge for our customers in the IC.

[JsizzleR]

Customer that would benefit from this issue: https://rstudioide.zendesk.com/agent/tickets/64039

[atheriel]

I believe this is a bug that we can't work around without upstream changes. The reproducible example by @aronatkins above is actually incorrect, but related. Building the same Docker image and running through the token generation code manually:

$ docker run --rm --entrypoint=bash -it rsconnect-fips
[root@0d82b833db72 /]# R --silent
> key <- openssl::rsa_keygen(2048) 
> priv.der <- openssl::write_der(key) 
> pub.der <- openssl::write_der(key$pubkey) 
Error: OpenSSL error in EVP_DigestInit_ex: disabled for fips
> traceback()
11: rawhash(x, algo, key)
10: rawstringhash(x, "md5", key)
9: md5(fp)
8: as.list.pubkey(pubkey)
7: as.list(pubkey)
6: as.list.key(x)
5: as.list(x)
4: `$.key`(k, pubkey)
3: k$pubkey
2: der_export(x)
1: openssl::write_der(k$pubkey)

This indicates that when the openssl packages generates DER for a public key it will (1) actually generate the public key on the fly from the raw bytes of the private key; and (2) in doing so compute a fingerprint, which is hardcoded to use MD5 (which fails due to FIPS).

A related issue is also the cause of the reproducible failure posted above:

> openssl::rsa_keygen(2048)
Error: OpenSSL error in EVP_DigestInit_ex: disabled for fips
> traceback()
7: rawhash(x, algo, key)
6: rawstringhash(x, "md5", key)
5: hashfun(unlist(unname(hashdata)))
4: fingerprint.pubkey(pk)
3: fingerprint(pk)
2: print.key(x)
1: (function (x, ...) 
   UseMethod("print"))(x)

That is, printing a key will also call some code that generates a nice, user-friendly, hardcoded-to-MD5 fingerprint of the key.

In sum: the openssl package hardcodes MD5 hashes in a few places. On FIPS-compliant systems, these calls will always fail. I think some upstream changes are warranted in this case, I'll try to submit some PRs and see what the response is like.

[atheriel]

This has been fixed in the upstream openssl package, now on CRAN at version 2.0.0. I've tested on both RHEL8 and CentOS7:

> openssl::fips_mode()
[1] TRUE
> rsconnect:::generateToken()
<omitted>

I'll submit a PR bumping our Import to 2.0.0, which should fix this for users on a fresh install.