Configuration check error message

[tomotsuguseki]

I encountered following error message in configuration check in AWS.

File upload directory is NOT writable on Amazon S3 server - CRITICAL: It is HIGHLY recommended that you investigate your Amazon S3 connection info on the File Upload Settings page in the Control Center. REDCap is not able to successfully store files in the Amazon S3 bucket named "$amazon_s3_bucket". Please make sure all the connection values are correct and also that the directory is writable. Some functionality within REDCap will not be functional until files can be written to that bucket.

File Upload Settings in control center seems fine. Could you please give me some advice to fix the problem?

[tomotsuguseki]

I encountered the same error in #23. Excerpt in redcapConfig.sql seems like this. Something incorrect?

-- S3 Integration
UPDATE redcap_config set value = '2' where field_name = 'edoc_storage_option';
UPDATE redcap_config SET value = 'APPLICATION_BUCKET_NAME' WHERE field_name = 'amazon_s3_bucket';
UPDATE redcap_config SET value = 'REDCAP_IAM_USER_ACCESS_KEY' WHERE field_name = 'amazon_s3_key';
UPDATE redcap_config SET value = 'REDCAP_IAM_USER_SECRET' WHERE field_name = 'amazon_s3_secret'; 
UPDATE redcap_config SET value = 'REGION' WHERE field_name = 'amazon_s3_endpoint'; 

[jl4nz]

I see... Are you using this code version 1.0.6 or latest pull from main branch? In 1.0.6 we did some fixes on this.

[jl4nz]

Your redcapConfig.sql s3 section seems correct.

[jl4nz]

For new deployments this should work correctly, however, to unblock the situation you could configure this manually.

The access key and secret are in your account AWS Secrets Manager with the name including REDCaps3accessSecret, you can copy the secret values and update in REDCap

Example:

You could also check the Application logs in ECS / AppRunner to see how the database values are configured.

1. If REDCap initial settings already configured, skipping is in the logs you are using the correct AccessKey for S3
2. If Using provided redcapConfig.sql for REDCap settings and Configuring REDCap DB settings... means the configuration script used the S3 secret and updated the database values.

https://github.com/aws-samples/redcap-on-aws/blob/main/containers/redcap-docker-apache/scripts/redcap_configure.sh#L84

DB_S3_ACCESS_KEY=$(mysql redcap -h ${RDS_HOSTNAME} -u ${RDS_USERNAME} -p${RDS_PASSWORD} -se "select value from redcap_config where field_name='amazon_s3_key'")

if [ "$DB_S3_ACCESS_KEY" = "$S3_ACCESS_KEY" ]; then
    echo '- REDCap initial settings already configured, skipping'
else
    # REDCap SQL Initialization Configuration
    REDCAP_CONFIG_SQL=/etc/redcap-entry/redcapConfig.sql

    if [ -f "$REDCAP_CONFIG_SQL" ]; then
        echo " - Using provided redcapConfig.sql for REDCap settings"
    else
        echo " - Using default REDCap DB settings"
        REDCAP_CONFIG_SQL=/etc/redcap-entry/redcapConfig.default.sql
    fi

    echo ' - Configuring REDCap DB settings...'
    sed -i "s|\\APPLICATION_BUCKET_NAME|$S3_BUCKET|g; s|\\REDCAP_IAM_USER_ACCESS_KEY|${S3_ACCESS_KEY//\//\\/}|g; s|\\REDCAP_IAM_USER_SECRET|${S3_SECRET_ACCESS_KEY//\//\\/}|g; s|\\REGION|$AWS_REGION|g;" $REDCAP_CONFIG_SQL
    mysql -h ${RDS_HOSTNAME} -u ${RDS_USERNAME} -D redcap --password=${RDS_PASSWORD} <$REDCAP_CONFIG_SQL
    echo ' - Done'
fi

[tomotsuguseki]

Thank you for your reply. Application logs said that Using provided redcapConfig.sql for REDCap settings and Configuring REDCap DB settings.... And redcap_configure.sh seems the same as you provided. What's the next step?

[jl4nz]

Thanks, so it's trying to configure a new bucket or S3Config value that is different from the secret or database current setting.

1. Can you check if the AWS Access key in REDCap configuration matches the secret value stored in AWS Secrets manager?

2. Also, in the secret value the bucket name is a bucket that exists?

The other option is to deploy the temporary deploy the EC2 instance to connect to the database and inspect what value is getting set. https://github.com/aws-samples/redcap-on-aws?tab=readme-ov-file#1-ec2-server-stack, after that you can use the command like aws ssm start-session --target i-<instance_id> --region ap-northeast-1 --profile redcap to connect the terminal. Part of this command is printed in the console, ssmPortForward: after deployment.

[jl4nz]

If you want to connect the database from the temporary EC2 instance you can do:

Terminal

 aws ssm start-session --target i-<instance_id>  --region <aws_region> --profile <aws_profile>

In the instance:

sudo su
wget https://dev.mysql.com/get/mysql80-community-release-el9-1.noarch.rpm 
dnf install mysql80-community-release-el9-1.noarch.rpm -y
rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2023
dnf install mysql-community-client -y
aws secretsmanager get-secret-value --secret-id  $DB_SECRET_ID

This will list the SecretString to connect, use this data to create the mysql connect command:

mysql -h <host> -u <user> -p

REDCAP DATABASE mysql> use redcap mysql> select * from redcap_config where field_name = 'amazon_s3_key';

You could update this as required to an empty value, deploy again, and then it should update with the correct keys

[jl4nz]

The other option is that this can be failing due the secret key has an escape character and the bash script is only inserting part of the secret.

Checking the UI or database agains the AWS secrets manager stores value will clear this.

Tomorrow I’ll investigate the possibility of the escape character

[tomotsuguseki]

Thanks for your effort. Bucket name was the same with that in the S3, but I couldn't find information in Secrets manager. I destroyed & rebuilt the REDCap again, but the result was the same. I look forward to your progress.

[jl4nz]

So after some investigation the Secret access key is an alpha numeric value with these possible characters / + =.

We handle this with sed, but we will update the code to a version that is easier to read. You can try this in the meantime for the redcap_configure.sh

redcap_configure.sh

#!/bin/bash

# Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: LicenseRef-.amazon.com.-AmznSL-1.0
# Licensed under the Amazon Software License  http://aws.amazon.com/asl/

set -e

DB_SECRET="$(aws secretsmanager get-secret-value --secret-id ${DB_SECRET_ID} --query 'SecretString' --output text --region ${AWS_REGION})"
DB_SALT="$(aws secretsmanager get-secret-value --secret-id ${DB_SALT_SECRET_ID} --query 'SecretString' --output text --region ${AWS_REGION})"
SES_CREDENTIALS="$(aws secretsmanager get-secret-value --secret-id ${SES_CREDENTIALS_SECRET_ID} --query 'SecretString' --output text --region ${AWS_REGION})"
S3_SECRET="$(aws secretsmanager get-secret-value --secret-id ${S3_SECRET_ID} --query 'SecretString' --output text --region ${AWS_REGION})"

export RDS_HOSTNAME="$(echo $DB_SECRET | jq -r .host)"
export RDS_USERNAME="$(echo $DB_SECRET | jq -r .username)"
export RDS_PASSWORD="$(echo $DB_SECRET | jq -r .password)"
export RDS_DBNAME="$(echo $DB_SECRET | jq -r .dbname)"
export RDS_PORT="$(echo $DB_SECRET | jq -r .port)"

export SES_USERNAME="$(echo $SES_CREDENTIALS | jq -r .username)"
export SES_PASSWORD="$(echo $SES_CREDENTIALS | jq -r .password)"

export S3_ACCESS_KEY="$(echo $S3_SECRET | jq -r .AccessKeyId)"
export S3_SECRET_ACCESS_KEY="$(echo $S3_SECRET | jq -r .SecretAccessKey)"

# DB_SALT in alphanumeric
export DB_SALT_ALPHA=$(echo -n "$DB_SALT" | sha256sum | cut -d' ' -f1)

if [ "$USE_IAM_DB_AUTH" = 'true' ]; then
    echo "- Using IAM auth"
    mysql -h ${RDS_HOSTNAME} -u ${RDS_USERNAME} -D ${RDS_DBNAME} --password=${RDS_PASSWORD} -e "
        CREATE USER IF NOT EXISTS 'redcap_user'@'%' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
        ALTER USER IF EXISTS 'redcap_user'@'%' IDENTIFIED WITH AWSAuthenticationPlugin AS 'RDS';
        GRANT ALL PRIVILEGES ON \`${RDS_DBNAME}\`.* TO 'redcap_user'@'%';
        GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'redcap_user'@'%';
        FLUSH PRIVILEGES;
    "
    echo "include '/usr/local/share/redcap/redcap_connect_iam.php';" >>/var/www/html/database.php
else
    echo "- Using base auth"
    echo "include '/usr/local/share/redcap/redcap_connect_base.php';" >>/var/www/html/database.php
fi

# EMAIL Setting for REDCap
if [ -z "$SES_USERNAME" ]; then
    echo "Credentials not set, smtp will not be configured"
else
    cat <<EOF >/etc/msmtprc
defaults
tls on
tls_starttls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
syslog on

account default
host email-smtp.${AWS_REGION}.amazonaws.com
port 587
auth on
user $SES_USERNAME
password $SES_PASSWORD
from ${SMTP_EMAIL:=localhost}
EOF
fi

# REDCap initial DB SETUP
if [ -f "/var/www/html/install.php" ]; then
    echo " - Executing install.php"
    cd /var/www/html
    php -r '$_GET["auto"]=1; $_GET["sql"]=1 ; $_SERVER["REQUEST_METHOD"] = "POST"; $_SERVER["PHP_SELF"]= "install.php"; require_once("install.php");'
fi

# REDCap replica config for Aurora serverless V2
if [ ! -z "$READ_REPLICA_HOSTNAME" ]; then
    echo "- Using replica"
    echo "include '/usr/local/share/redcap/redcap_connect_replica.php';" >>/var/www/html/database.php
    mysql -h ${RDS_HOSTNAME} -u ${RDS_USERNAME} -D ${RDS_DBNAME} --password=${RDS_PASSWORD} -e "
        UPDATE IGNORE redcap_config SET value = '1' WHERE field_name = 'read_replica_enable';
    "
    REDCAP_VERSION=$(ls /var/www/html/ -1 | grep -E "^redcap_v") 
    # Disable the lag check as this does not apply to Amazon RDS Aurora Serverless V2
    sed -i 's/$bypassReadReplicaLagCheck=false)/$bypassReadReplicaLagCheck=true)/g' /var/www/html/$REDCAP_VERSION/Config/init_functions.php
fi

DB_S3_ACCESS_KEY=$(mysql redcap -h ${RDS_HOSTNAME} -u ${RDS_USERNAME} -p${RDS_PASSWORD} -se "select value from redcap_config where field_name='amazon_s3_key'")

# Force new database configuration
# DB_S3_ACCESS_KEY='key'

if [ "$DB_S3_ACCESS_KEY" = "$S3_ACCESS_KEY" ]; then
    echo '- REDCap initial settings already configured, skipping'
else
    # REDCap SQL Initialization Configuration
    REDCAP_CONFIG_SQL=/etc/redcap-entry/redcapConfig.sql

    if [ -f "$REDCAP_CONFIG_SQL" ]; then
        echo " - Using provided redcapConfig.sql for REDCap settings"
    else
        echo " - Using default REDCap DB settings"
        REDCAP_CONFIG_SQL=/etc/redcap-entry/redcapConfig.default.sql
    fi

    echo ' - Configuring REDCap DB settings...'
    ESCAPED_S3_SECRET_ACCESS_KEY=$(printf '%s\n' "$S3_SECRET_ACCESS_KEY" | sed -e 's/[\/&]/\\&/g')
    sed -i "s/APPLICATION_BUCKET_NAME/$S3_BUCKET/g" $REDCAP_CONFIG_SQL
    sed -i "s/REDCAP_IAM_USER_ACCESS_KEY/$S3_ACCESS_KEY/g" $REDCAP_CONFIG_SQL
    sed -i "s/REDCAP_IAM_USER_SECRET/$ESCAPED_S3_SECRET_ACCESS_KEY/g" $REDCAP_CONFIG_SQL
    sed -i "s/REGION/$AWS_REGION/g" $REDCAP_CONFIG_SQL
    mysql -h ${RDS_HOSTNAME} -u ${RDS_USERNAME} -D redcap --password=${RDS_PASSWORD} <$REDCAP_CONFIG_SQL
    echo ' - Done'
fi

exec "$@"

You can uncomment the 87th line, # DB_S3_ACCESS_KEY='key' to force the update of the database.

Run yarn deploy... and then the aws lambda invoke...

Comment the 87th line again to avoid further updates.

[tomotsuguseki]

Thank you for your modification. This file worked well.