Fixing Open On Demand "Bad record MAC" error

[marcodelapierre]

Hi,

I just wanted to document the steps I take to successfully get started with the tutorial using the default container images on Docker Hub.

Preliminary:

• Clone the repo:

  git clone git@github.com:ubccr/hpc-toolset-tutorial.git

• Spin up the Docker Compose deployment:

  cd hpc-toolset-tutorial
  ./hpcts start

• Login to the frontend node via SSH (credentials in Accessing the Applications):

  ssh -p 6222 hpcadmin@localhost

After this,I would be able to successfully connect to Coldfront and XDMod via web browser as outlined in Accessing the Applications.

However, attempting to open the Open On Demand web portal would result in an error:

https://localhost:3443/

Error on web browser: 500 Internal Server Error.

Error via Docker Compose logs:

docker compose logs ondemand

ondemand  | 2024/11/06 06:22:58 http: TLS handshake error from 127.0.0.1:38788: local error: tls: bad record MAC

The solution was to update the certificates inside the Open On Demand container, and then restart it, as documented in https://discourse.openondemand.org/t/bad-record-mac-with-hpc-tools-tutorial/3790 .

This is the script that made it, as adapted from the link above, and run in a shell terminal on the host machine:

#!/bin/bash

# Generate CA
openssl genrsa -out ca.key 4096
openssl req -new -x509 -days 100000 -sha256 -key ca.key -extensions v3_ca -out ca.crt -subj "/C=US/ST=YourState/L=YourCity/O=YourOrganization/OU=YourDepartment/CN=localhost"
# Generate certificate request
openssl genrsa -out localhost.key 2048
openssl req -new -sha256 -key localhost.key -out localhost.csr -subj "/C=US/ST=YourState/L=YourCity/O=YourOrganization/OU=YourDepartment/CN=localhost"
# Config for signing cert
cat > localhost.ext << EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = DNS:localhost
extendedKeyUsage = serverAuth
EOF

# Sign cert request and generate cert
openssl x509 -req -CA ca.crt -CAkey ca.key -CAcreateserial \
  -in localhost.csr -out localhost.crt \
  -days 100000 -sha256 -extfile localhost.ext

#Add to Docker Container
docker cp ca.key ondemand:/etc/pki/tls/
docker cp ca.crt ondemand:/etc/pki/tls/
docker cp localhost.key ondemand:/etc/pki/tls/private/
docker cp localhost.csr ondemand:/etc/pki/tls/certs/
docker cp localhost.ext ondemand:/etc/pki/tls/
docker cp localhost.crt ondemand:/etc/pki/tls/certs/
docker cp ca.crt ondemand:/etc/pki/ca-trust/source/anchors/

#Update cert trust in Docker Container
docker exec ondemand update-ca-trust extract

#Restart Docker Container
docker restart ondemand

After executing it, I can successfully access the Open On Demand web portal on https://localhost:3443 .

I hope this can help!