boozallen / aissemble

Booz Allen's lean manufacturing approach for holistically designing, developing and fielding AI solutions across the engineering lifecycle from data processing to model building, tuning, and training to secure operational deployment
Other
34 stars 8 forks source link

Feature: Write build caches to S3 #461

Closed ewilkins-csi closed 2 weeks ago

ewilkins-csi commented 2 weeks ago

Description

In #429 we started saving the Docker build cache during the build workflow. We discussed that the size of the Docker cache may end up causing issues and it has. Because it's ~9GB, the docker cache will often cause all other caches to be purged and, at least once, has failed to save seemingly due to its size. In addition, it takes about 9.75 mins to save the cache to GH caches. We should go ahead and implement the Runs-On S3 fast cache action to get around these issues.

DOD

Test Strategy/Script

  1. View the latest build and verify the cache actions are from the runs-on/cache action
  2. Confirm the caches saved to AWS are < 1 day old
ewilkins-csi commented 2 weeks ago

DoD/OTS with @csun-cpointe and @carter-cundiff

csun-cpointe commented 2 weeks ago

Test passed! Confirm the caches saved to AWS are < 1 day old Screenshot 2024-11-08 at 11 44 28 AM Screenshot 2024-11-08 at 11 44 06 AM Screenshot 2024-11-08 at 11 43 45 AM Screenshot 2024-11-08 at 11 43 33 AM Screenshot 2024-11-08 at 11 40 40 AM

View the latest build and verify the cache actions are from the runs-on/cache action

Screenshot 2024-11-08 at 11 38 33 AM Screenshot 2024-11-08 at 11 37 43 AM Screenshot 2024-11-08 at 11 37 32 AM Screenshot 2024-11-08 at 11 37 00 AM Screenshot 2024-11-08 at 11 36 53 AM