boozallen / aissemble

Booz Allen's lean manufacturing approach for holistically designing, developing and fielding AI solutions across the engineering lifecycle from data processing to model building, tuning, and training to secure operational deployment
Other
33 stars 8 forks source link

Improve downstream projects to more easily specify their project specific docker repository #75

Closed colinpalmer-pro closed 3 months ago

colinpalmer-pro commented 4 months ago

Description

In aiSSEMBLE downstream projects, there is currently no defined default docker repository for publishing and retrieving images. In order to make project generation as smooth as possible, we want the functionality to specify the docker repository during project creation and make it straightforward to update the repo.

Definition of Done

Test Steps

Downstream Project

cd issue-75

- Run the following command to validate that the private docker repository was configured into the project's `docker.project.repository.url` variable:
  - `% cat pom.xml | grep private-docker-base-url`
  - Validate the response is:
    - `<docker.project.repository.url>private-docker-base-url/my-docker-repo/</docker.project.repository.url>` 
- Run the following command to validate that the private docker repository url is referenced in the `ci` profile in the `issue-75-docker/pom.xml`
  - `% cat issue-75-docker/pom.xml`
  - Near the bottom of the output in the `profiles` tag, confirm there is a `ci` profile with the `orphedomos-maven-plugin` and it has the `repoUrl` tag e.g. `<repoUrl>${docker.project.repository.url}</repoUrl>`
- Create a json file, `SimpleDataDeliveryExample.json` in `issue-75/issue-75-pipeline-models/src/main/resources/pipelines` and add the following contents.

{ "name":"SimpleDataDeliveryExample", "package":"com.boozallen.aissemble.documentation", "type":{ "name":"data-flow", "implementation":"data-delivery-spark", "executionHelpers": ["airflow"] }, "steps":[ { "name":"IngestData", "type":"synchronous", "dataProfiling":{ "enabled":false } } ] }


- Build the project from the root - `mvn clean install`
  - Follow all of the manual actions to add to the project and keep building until there are no more instructions (this will take a few builds)
  - When there are no more manual actions, run the build one more time without the cache (`mvn clean install -Dmaven.build.cache.enabled=false`. Noticed that some manual instructions were not shown with cache enabled.
- From the root of the project, run `tilt up`
  - Validate that all resources come up successfully (specifically `policy-decision-point`, `vault`, `spark-worker-image` and `airflow`)
  - Run `kubectl get pods` to show that all pods are either running or completed
- Press Ctrl+c and `tilt down` to bring the application down
  - In `issue-75-deploy/src/main/resources/apps/policy-decision-point/values.yaml`, ensure you see the line: `dockerRepo: private-docker-base-url/my-docker-repo/`
  - In `issue-75-deploy/src/main/resources/apps/airflow/values.yaml`, ensure you see the line: `defaultAirflowRepository: private-docker-base-url/my-docker-repo/issue-75-deploy-airflow-docker`

#### Documentation
- Navigate to the aissemble docs [Creating a Project](https://boozallen.github.io/aissemble/aissemble/current/archetype.html) page and validate that instructions for `dockerProjectRepositoryUrl` exist
- Validate that the [release notes for 1.7.0](https://github.com/boozallen/aissemble/blob/dev/DRAFT_RELEASE_NOTES.md) have instructions to manually update the docker project repository under the "Update Docker Repository Configuration" heading
colinpalmer-pro commented 3 months ago

Test steps pass, closing issue.