Booz Allen's lean manufacturing approach for holistically designing, developing and fielding AI solutions across the engineering lifecycle from data processing to model building, tuning, and training to secure operational deployment
Other
33
stars
8
forks
source link
Improve downstream projects to more easily specify their project specific docker repository #75
In aiSSEMBLE downstream projects, there is currently no defined default docker repository for publishing and retrieving images. In order to make project generation as smooth as possible, we want the functionality to specify the docker repository during project creation and make it straightforward to update the repo.
Definition of Done
[x] Update archetype to allow specification of project-specific Docker repository
[x] Ensure that project-specific Docker repository is enabled for project-specific modules
[x] Update documentation to highlight how to specify a project-specific Docker repository
[x] Update release notes to manually add a private Docker repository for projects migrating from a prior version of aissemble
Test Steps
Downstream Project
Make sure the dev branch is up-to-date and build the project (mvn clean install)
Run the archetype plugin with the following commands to create a downstream project:
- Run the following command to validate that the private docker repository was configured into the project's `docker.project.repository.url` variable:
- `% cat pom.xml | grep private-docker-base-url`
- Validate the response is:
- `<docker.project.repository.url>private-docker-base-url/my-docker-repo/</docker.project.repository.url>`
- Run the following command to validate that the private docker repository url is referenced in the `ci` profile in the `issue-75-docker/pom.xml`
- `% cat issue-75-docker/pom.xml`
- Near the bottom of the output in the `profiles` tag, confirm there is a `ci` profile with the `orphedomos-maven-plugin` and it has the `repoUrl` tag e.g. `<repoUrl>${docker.project.repository.url}</repoUrl>`
- Create a json file, `SimpleDataDeliveryExample.json` in `issue-75/issue-75-pipeline-models/src/main/resources/pipelines` and add the following contents.
- Build the project from the root - `mvn clean install`
- Follow all of the manual actions to add to the project and keep building until there are no more instructions (this will take a few builds)
- When there are no more manual actions, run the build one more time without the cache (`mvn clean install -Dmaven.build.cache.enabled=false`. Noticed that some manual instructions were not shown with cache enabled.
- From the root of the project, run `tilt up`
- Validate that all resources come up successfully (specifically `policy-decision-point`, `vault`, `spark-worker-image` and `airflow`)
- Run `kubectl get pods` to show that all pods are either running or completed
- Press Ctrl+c and `tilt down` to bring the application down
- In `issue-75-deploy/src/main/resources/apps/policy-decision-point/values.yaml`, ensure you see the line: `dockerRepo: private-docker-base-url/my-docker-repo/`
- In `issue-75-deploy/src/main/resources/apps/airflow/values.yaml`, ensure you see the line: `defaultAirflowRepository: private-docker-base-url/my-docker-repo/issue-75-deploy-airflow-docker`
#### Documentation
- Navigate to the aissemble docs [Creating a Project](https://boozallen.github.io/aissemble/aissemble/current/archetype.html) page and validate that instructions for `dockerProjectRepositoryUrl` exist
- Validate that the [release notes for 1.7.0](https://github.com/boozallen/aissemble/blob/dev/DRAFT_RELEASE_NOTES.md) have instructions to manually update the docker project repository under the "Update Docker Repository Configuration" heading
Description
In aiSSEMBLE downstream projects, there is currently no defined default docker repository for publishing and retrieving images. In order to make project generation as smooth as possible, we want the functionality to specify the docker repository during project creation and make it straightforward to update the repo.
Definition of Done
Test Steps
Downstream Project
mvn clean install
)cd issue-75
{ "name":"SimpleDataDeliveryExample", "package":"com.boozallen.aissemble.documentation", "type":{ "name":"data-flow", "implementation":"data-delivery-spark", "executionHelpers": ["airflow"] }, "steps":[ { "name":"IngestData", "type":"synchronous", "dataProfiling":{ "enabled":false } } ] }