We've been installing AWS CLI v1 on the analysis container using the "download the bundle and run the install script" method, but that no longer supports older versions of Python. Rebecca emailed with a crash she got when building the analysis container:
Rather than have to upgrade the version of Python we're running in the container, we can just switch to AWS CLI v2, which has no external Python dependency. The analysis only uses the AWS CLI for simple S3 uploading and downloading, so moving to v2 doesn't require any changes to the commands we're running.
Notes
I also changed python to python3 in the list of apt-get packages to install. It was already getting pulled in by python3-dev, but I don't think there's any reason to keep the Python 2 package around, and if there's a place that we're accidentally still using Python 2, I'd rather find out about it than let it keep quietly working.
We upgraded to AWS CLI v2 in the VM then reverted back to v1 because we couldn't use v2 on the CI server. This is independent of that, and doesn't suffer the same compatibility problems since it's inside the analysis container.
Testing Instructions
Either build everything from scratch (i.e. destroy your VM and run scripts/setup) or build just the analysis container from scratch by starting the VM, sshing to it, and running docker-compose build --pull --no-cache analysis
Run a test analysis job to confirm that the steps that download from and upload to S3 work properly
Overview
We've been installing AWS CLI v1 on the analysis container using the "download the bundle and run the install script" method, but that no longer supports older versions of Python. Rebecca emailed with a crash she got when building the analysis container:
Rather than have to upgrade the version of Python we're running in the container, we can just switch to AWS CLI v2, which has no external Python dependency. The analysis only uses the AWS CLI for simple S3 uploading and downloading, so moving to v2 doesn't require any changes to the commands we're running.
Notes
pythontopython3in the list of apt-get packages to install. It was already getting pulled in bypython3-dev, but I don't think there's any reason to keep the Python 2 package around, and if there's a place that we're accidentally still using Python 2, I'd rather find out about it than let it keep quietly working.Testing Instructions
scripts/setup) or build just the analysis container from scratch by starting the VM, sshing to it, and runningdocker-compose build --pull --no-cache analysisChecklist