Feature: replace aissemble-airflow 1.7.0 references with community docker image

[d-ryan-ashcraft]

Description

The aissemble-airflow docker image had minimal impact and was removed as part of feature #221. We purposely left helm references to the aissemble 1.7.0 container until this ticket migrates this reference to an appropriate community chart and validates that it also still works.

DOD

• [x] Replace aissemble-airflow docker image references with the chosen community replacement (airflow).
  • Ensure the replacement does NOT break the build
• [x] Develop migration script for existing Dockerfile that are NOT using community airflow docker image.
• [x] Update release notes indicating airflow docker image has been migrated over to the community docker image.

Test Strategy/Script

Test 1: Ensure downstream project that is using airflow builds the community docker image.

• Step 1: Generate a new downstream project that uses Airflow by running the following: mvn archetype:generate -DarchetypeGroupId=com.boozallen.aissemble -DarchetypeArtifactId=foundation-archetype -DarchetypeVersion=1.9.0-SNAPSHOT -DgroupId=com.airflowtest -DartifactId=airflowtest -DprojectGitUrl=url -DprojectName=airflowtest && cd airflowtest
• Step 2: Add SparkPipeline.json to src/main/resources/pipelines (file is attached).
• Step 3: Run mvn clean install and resolve all the manual actions.
• Step 4: Deploy the project locally by running tilt up; tilt down.
• Step 5: Wait till all the Airflow pods are up and running. Once they are, go to the airflow-scheduler pod and verify the Dockerfile is pulling the community Docker image (FROM apache/airflow:2.9.3).
• Step 6: Navigate to localhost:9085 and verify the Airflow UI is reachable.
• Step 7: Log into Airflow (default username: admin; default password: aiops).
• Step 8: Verify the version has been updated to Version: 2.9.3 (located on the bottom left corner).

Test 2: Upgrade downstream project versions to ensure migration works.

• Step 1: Download the attached Airflow downstream project labeled airflowtest2.zip. This is a project that is on 1.8.0.
• Step 2: Navigate to the root pom and upgrade the aiSSEMBLE version to 1.9.0-SNAPSHOT.
• Step 3: Run the build mvn clean install until all manual actions have been resolved.
  • Note: it's not necessary there will be any.
  • Step 4: Deploy the project locally by running tilt up; tilt down.
  • Step 5: Wait till all the Airflow pods are up and running. Once they are, go to the airflow-scheduler pod and verify the Dockerfile is pulling the community Docker image (FROM apache/airflow:2.9.3).
  • Step 6: Navigate to localhost:9085 and verify the Airflow UI is reachable.
  • Step 7: Log into Airflow (default username: admin; default password: aiops).
  • Step 8: Verify the version has been updated to Version: 2.9.3 (located on the bottom left corner).

References/Additional Context

• Docker Image for Apache Airflow

[habibimoiz]

DoD Questions/Clarifications:

• Do we want to upgrade the airflow Helm chart as well? We are currently on 1.10.0, latest is 1.15.0. Here you can find all the changes included with the version upgrade: Airflow 1.15.0 Release Notes
  • We want to upgrade the chart to 1.15.0.
• Do we want to default the Airflow docker image to "latest" or use a static version? (currently the default docker image is 2.9.3)
  • We want to use the default tag of 2.9.3.
• Outside of the Dockerfile, is there anything else that needs to be migrated as part of this issue?
  • migrate the Dockerfile, thats all.
• Any recommended metamodel to test the switch over to the airflow community docker image?

[habibimoiz]

DoD reviewed and approved by @ewilkins-csi.

[habibimoiz]

Currently running into an issue when tilting up. persistentvolumeclaim "model" not found` on the following airflow pods.

• airflow-run-airflow-migrations
• airflow-create-user
• airflow-webserver
• airflow-scheduler

[habibimoiz]

Testing files.

SparkPipeline.json airflowtest2.zip

[habibimoiz]

OTS completed w/ @chang-annie.