cwbi-s3-static-appserver hot reload of new content not working as expected in ECS/Fargate deployment

brettpalmberg commented 3 years ago

cwbi-s3-static-appserver: The sidecar container and appserver container share a volume, mounted at /data.

sidecar is responsible for an s3 sync on a regular interval, to keep files in the container volume current with what is available in the s3 bucket (see entrypoint.sh)

appserver performs a graceful restart on a regular specified interval with the intent that new content sync'd by sidecar to directory /data is served. This appears to work locally. New files and changes to files are not served by appserver with current deployment. Logs show that (1) sidecar is regularly syncing the s3 bucket files with the local volume and (2) the graceful restart is happening; however, new content is not displayed when a change is made to contents of the s3 bucket being served.

This is currently affecting apps being served from buckets:

cwbi-apps-develop
cwbi-apps-stable

brettpalmberg commented 3 years ago

Update

Hot reload is working. This was confirmed by adding a new file to s3 bucket cwbi-apps-develop at key /cumulus/index.html. The sidecar and appserver automatically begin serving the file. The file index.html was then updated and the updated file was then served. Therefore ... something else is at play. Next steps: Investigate how files are copied by CI/CD process by github.com/USACE/instrumentation-ui to bucket cwbi-apps-develop and if file changes are detected in webpack build by aws s3 sync in sidecar container.

Troubleshooting continues ...

brettpalmberg commented 3 years ago

Solved

This is one of those issues that takes a good amount of time to debug, but only takes a few keystrokes to fix.

TLDR

aws s3 sync by default does not copy files from s3 to local when a filename already exists on local, the file size has not changed, and s3 is newer than local. If a file on s3 is changed, but the change is such that filesize does not change - aws s3 sync will not download the modified file. The filesize of index.html as built by webpack remains the same from build to build so updated index.html files are never copied from s3 to local to be served by appserver unless the option --exact-timestamps is used with aws s3 sync.

Detailed Explanation

Hot reloading was being tested by making a small change to a React.js application's landing page - i.e. adding TEST MESSAGE THURSDAY APR08 and checking the hosting url to watch for the change to show up (see image below). However, the change never shows up unless the Fargate task - i.e. appserver and sidecar containers - were completely restarted.

How new files arrive in webroot to be served by appserver

To add the text TEST MESSAGE THURSDAY APR08 to the landing page, we add <h3>TEST MESSAGE THURSDAY APR08</h3> to a file in our repository and push changes to github. When changes are pushed, CI/CD takes over, creates a new build with npm run build and webpack creates a directory of files (see below). This is the directory that is served by a webserver.

Let's explore two files in our new build, main.553bbcf6.chunk.js and index.html.

main.553bbcf6.chunk.js

This is one files that changed in the webpack build. Look, there's our test message in the image above! This file needs to be served if we are going to see our test message on the landing page.

index.html

This is the file that is served by appserver when we go to the main url for our application. It points to the javascript that is our app, main.553bbcf6.chunk.js.

So far so good - CI/CD copies the build directory and all files to an S3 bucket, let's call the bucket cwbi-apps-develop.

Next let's look at how the files that make-up our application (stored in a directory on S3) come to be served by appserver.

Appserver does not serve files directly from S3, but rather from a container volume mounted at directory /data. Directory /data is populated from and kept in sync with the most up-to-date files coming from CI/CD in S3 bucket cwbi-apps-develop. This is done using a lightweight sidecar container that calls aws s3 sync on a regular interval. Here is a simplified version showing what sidecar does:

while true
do
    echo "Sync With Bucket ${S3_BUCKET}; $(date)"
    aws s3 sync s3://${S3_BUCKET} /data/
    sleep 120
done

Let's reproduce what happens inside the container by calling aws s3 sync s3://cwbi-apps-develop /test_data_directory/ on our computer. First - let's populate /test_data_directory with the previous build of the application (without the message TEST MESSAGE THURSDAY APR08) so it's exactly replicating what the /data directory looks like inside the container. Second - let's run aws s3 sync and explore the results:

$ aws s3 sync s3://cwbi-apps-develop ./test_data_directory/ --dryrun
(dryrun) download: s3://cwbi-apps-develop/static/js/main.553bbcf6.chunk.js to test_server/static/js/main.553bbcf6.chunk.js
(dryrun) download: s3://cwbi-apps-develop/static/js/main.553bbcf6.chunk.js.map to test_server/static/js/main.553bbcf6.chunk.js.map

Notice ... aws s3 sync without additional command line options will copy the new chunk.js files (which include the new banner) to /data to be served; however, it will not copy the new index.html file that points to the new chunk.js file. Looking at the index.html files from the new and previous build alongside the documentation for aws s3 sync helps us see exactly what is going on.

Previous Build index.html

$ ls -al ./index.html && md5 ./index.html
-rw-r--r--  1 xxxxxx  2011572450  3818 Apr  8 11:26 index.html
MD5 (./index.html) = 5765a96cb68da398ab833d82f3935afb

File Content

New Build index.html

$ ls -al ./index.html && md5 ./index.html
-rw-r--r--  1 xxxxxx  2011572450  3818 Apr  7 19:55 index.html
MD5 (./index.html) = 14b9db4806c852493ad2a01b6858b5de

File Content

Observations

The Previous Build index.html and New Build index.html have the exact same filesize
Timestamp on file Previous Build index.html is older than New Build index.html stored on S3
The files are indeed different, confirmed by either an md5 hash of each or by examining file contents

From the aws s3 sync docs, we see that the command is working as expected. Without specifying the --exact-timestamps option, the newer index.html file is not copied to /data because it is named the same and has the exact same size as the file already present:

--exact-timestamps (boolean) When syncing from S3 to local, same-sized items will be ignored only when the timestamps match exactly. The default behavior is to ignore same-sized items unless the local version is newer than the S3 version.

In summary - the updated index.html file is never served by appserver because it never arrives in directory /data in the container. Furthermore, because the --delete option is not provided to aws s3 sync, both the index.html and chunk.js file from the previous build still exist on disk at /data and appserver happily serves them. From the outside, looking at the landing page, it looks like nothing has been updated.

Solution

Add --delete and --exact-timestamps options to aws s3 sync command here.

brettpalmberg commented 3 years ago

Fixed in commit here: https://github.com/USACE/cwbi-s3-static-appserver/commit/ad8067eb8cd9171b12ac6d4a33974cdbccb29875

USACE / instrumentation