Look into using containers rather than builds directly on the Jenkins build server.

[maxachis]

Building directly on the Jenkins build server poses a few problems, which using Docker Containers will help mitigate.

1. Different builds may have different environmental requirements, which may interfere with each other. While this can be partly mitigated with python virtual environments, it can't account for components which require ubuntu installations on the operating system itself. Docker will ensure each build remains in separate environments
2. In the event that something in a build goes catastrophically wrong (for example, a memory leak), the entire system could be affected. Docker containers reduce this risk by limiting the failure to within the container itself.
3. Testing builds can be a pain just because of the amount of manual input that is required. Using containers, with setups defined within their environments, will help avoid that.

Steps required

• [x] Follow steps in this Digital Ocean guide for setting up a CI/CD pipeline in Jenkins
• [x] Test out implementation of container solution in branch in Prod to Dev Migration Repo
• [x] Expand to other automation builds:
• [x] #226
• [x] automatic-archives (PR Ready)
• [x] data-sources-mirror (PR Ready)
• [ ] Add how-to notes to Notion

[maxachis]

Notes to eventually include in Notion:

Dockerfiles

What are Dockerfiles?

Dockerfiles are text files which run a series of commands in order to build a Docker image.

How we use Dockerfiles

Dockerfiles are used in the deployment of Jenkins builds, alongside Jenkinsfiles. Through Dockerfiles, we're able to ensure a consistent and repeatable environment that is contained in isolation from the rest of the environment. This enables more secure deployments as well as aids testing by helping to avoid the "Works On My Machine" problem.

In repositories which utilize Jenkins builds, a Dockerfile should be included in the root directory, along with a Jenkinsfile, containing the setup details for the requisite environment. Examples can be found in other repositories which are utilized in the Automation Manager.

Recommendations for Dockerfile implementation

• Follow Docker's own best practices guide
• Familiarize yourself with how Dockerfile caches build components. In brief: If a part of the Dockerfile or a file referenced by the Dockerfile doesn't change, that step can be re-used on future calls unless explicitly told otherwise, saving time. However, sometimes Docker will cache build steps even after they've been changed -- pay attention to the structure of your Dockerfile and test rigorously.
• For Python: Python installations can often be easily contained in a Docker image, most easily using the official Docker python images. However, in some cases, such as when using an Ubuntu image, extra steps are required which can be easily confused. Consider the below script as an example of one way (not necessarily the best) of installing within an Ubuntu image:

# Install dependencies necessary for add-apt-repository
RUN apt-get install software-properties-common -y
# Install Python and pip
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get update && apt-get install -y \
    python3.11 \
    python3-pip
# This section ensures that the Python package installation built inside the Dockerfile
# is accessible within the container.
COPY requirements.txt /opt/app/requirements.txt
WORKDIR /opt/app
RUN pip install --no-cache-dir -r requirements.txt --break-system-packages
COPY . /opt/app

Jenkins

[To fill in more later].

Environment variables

• [Discuss the plugin used to insert environment variables and how these can be passed into the docker container]