signatureHeaders sees OpenSSL error when FIPS enabled

[aronatkins]

When using FIPS in a Rocky Linux 9 container, calling openssl::signature_create can err like:

16.63 > rawsig <- openssl::signature_create(charToRaw(canonicalRequest), key = private_key)
16.63 Error in hash_sign(md, sk) :
16.63   OpenSSL error: 807014FC877F0000:error:03000098:digital envelope routines:evp_pkey_ctx_set_md:invalid digest:crypto/evp/pmeth_lib.c:959:
16.63 Calls: <Anonymous> -> hash_sign
16.63 Execution halted

This can happen when people attempt to configure a Posit Account using the RStudio IDE or using rsconnect::connectUser.

Folks encountering this problem can use rsconnect::connectApiUser as a workaround.

Here is the Dockerfile:

FROM rstudio/r-base:4.3-rockylinux9

RUN dnf update -y && dnf install -y openssl-devel
RUN fips-mode-setup --enable

COPY debugging.R /content/

RUN R -f /content/debugging.R

Here is debugging.R:

options(repos = c(CRAN = "https://cran.rstudio.com/"))

install.packages("digest")
install.packages("openssl")

# equivalent to rsconnect:::generateToken
generateToken <- function() {
  key <- openssl::rsa_keygen(2048L)
  priv.der <- openssl::write_der(key)
  pub.der <- openssl::write_der(key$pubkey)
  tokenId <- paste(c("T", openssl::rand_bytes(16)), collapse = "")

  list(
    token = tokenId,
    public_key = openssl::base64_encode(pub.der),
    private_key = openssl::base64_encode(priv.der)
  )
}

# equivalent to rsconnect:::rfc2616Date
rfc2616Date <- function(time = Sys.time()) {
  # set locale to POSIX/C to ensure ASCII date
  old <- Sys.setlocale("LC_TIME", "C")
  on.exit(Sys.setlocale("LC_TIME", old))

  strftime(Sys.time(), "%a, %d %b %Y %H:%M:%S GMT", tz = "GMT")
}

token <- generateToken()
path <- "/users/current"
method <- "GET"
date <- rfc2616Date()
md5 <- openssl::base64_encode(digest::digest("", algo = "md5", serialize = FALSE, raw = TRUE))
canonicalRequest <- paste(method, path, date, md5, sep = "\n")
private_key <- openssl::read_key(
     openssl::base64_decode(token$private_key),
     der = TRUE
)
# the signature_create line errs
rawsig <- openssl::signature_create(charToRaw(canonicalRequest), key = private_key)
signature <- openssl::base64_encode(rawsig)

With the Dockerfile and debugging.R files written into a directory, recreate the error with:

docker build .

[aronatkins]

A more minimal debugging.R that presents the same error:

options(repos = c(CRAN = "https://cran.rstudio.com/"))

install.packages("openssl")

key <- openssl::rsa_keygen(2048L)
rawsig <- openssl::signature_create(charToRaw("some data"), key = key)
signature <- openssl::base64_encode(rawsig)

[aronatkins]

Another variation on debugging.R..

key <- openssl::rsa_keygen(2048L)
priv_der <- openssl::write_der(key)
pub_der <- openssl::write_der(key$pubkey)
token <- list(
  public_key = openssl::base64_encode(pub_der),
  private_key = openssl::base64_encode(priv_der)
)
private_key <- openssl::read_key(
  openssl::base64_decode(token$private_key),
  der = TRUE
)
rawsig <- openssl::signature_create(charToRaw("some data"), key = key)
signature <- openssl::base64_encode(rawsig)

[hadley]

Related to #363?

[aronatkins]

The error can be avoided if we tell OpenSSL to allow SHA1 signatures:

ENV OPENSSL_ENABLE_SHA1_SIGNATURES=yes
RUN R -f /content/debugging.R

[aronatkins]

RHEL9 OpenSSL disables SHA1 signatures: https://gitlab.com/redhat/centos-stream/rpms/openssl/-/commit/78fb78d30755ae18fdaef28ef392f4e67c662ff6 (linked from https://github.com/VirusTotal/yara/issues/1864#issuecomment-1400252566)

[aronatkins]

Related to #363?

@hadley - Related, yes: That issue tracks an error when using MD5 to produce checksums; this one reports an error when using SHA1 to create signatures.

This particular SHA1 error was not previously seen - maybe because RHEL9 (with the added enforcement) is fairly recent.

[hadley]

Oh sorry, I meant #768

[aronatkins]

Oh sorry, I meant #768

Oh! I totally didn't find that issue. Yes, it looks to be the same error.

[aronatkins]

Support ticket reference: 91795

[rstub]

Support ticket reference: 96540

According to https://github.com/dotnet/runtime/issues/65874#issuecomment-1058187135 another workaround would be to set rh-allow-sha1-signatures in openssl.cnf.

[hadley]

Is the problem being tracked on the connect side too? Obviously it will need a change before we can use a different client side strategy.

[aronatkins]

Connect issue #14353 tracks shifting away from MD5; I did not find anything tracking this piece.

@mmarchetti - are you tracking a change to API keys in all workflows?

[christierney]

https://rstudioide.zendesk.com/agent/tickets/101890