department-of-veterans-affairs / va.gov-cms

Editor-centered management for Veteran-centered content.
https://prod.cms.va.gov
GNU General Public License v2.0
98 stars 69 forks source link

`content-build` self-hosted runners are built with insufficient disk space. #17069

Open ndouglas opened 8 months ago

ndouglas commented 8 months ago

Description

content-build runners are built with 120GB of disk space. However, an individual workflow run, depending on particulars, may take 80GB or more. Two workflows running at the same time may thus exhaust disk space and one or more of them may fail and leave the runner in a not-so-useful condition.

A short-term workaround for this is to manually run this workflow: https://github.com/department-of-veterans-affairs/devops/actions/workflows/packer-ghar.yml specifying ['content-build'] and 300 (GB) of disk space. However, this needs to be done around the 1st and 15th of each month, or else runners may fail.

This workflow for content-build might be extracted from the devops workflow, or the workflow file might be rewritten to better accommodate customizability for the runners.

Acceptance Criteria

gracekretschmer-metrostar commented 2 weeks ago

@timcosgrove has this work been completed?