boozallen / aissemble

Booz Allen's lean manufacturing approach for holistically designing, developing and fielding AI solutions across the engineering lifecycle from data processing to model building, tuning, and training to secure operational deployment
Other
34 stars 8 forks source link

Feature: Document the setup of custom GitHub action runners that can execute the aiSSEMBLE build. #467

Closed ewilkins-csi closed 1 week ago

ewilkins-csi commented 1 week ago

Description

Our build has been failing recently due to disk pressure on the cluster that hosts our GHA runners. During the process of expanding the disk size for the cluster, the ARC deployment responsible for provisioning runners on-demand for aiSSEMBLE stopped working. It has been restored, but we should document the setup process and any caveats so that we can easily redeploy ARC for aiSSEMBLE if/when we need to.

DOD

Test Strategy/Script

References/Additional Context

We could create a chart for deploying ARC in a compatible way, but the focus of this ticket is largely to get the CI build working again so we can continue with normal operations.

ewilkins-csi commented 1 week ago

DoD/OTS with @csun-cpointe

csun-cpointe commented 1 week ago

Test passed! ARC ReadMe Screenshot 2024-11-14 at 10 56 50 AM

Latest Build Success Screenshot 2024-11-14 at 10 57 28 AM