Significantly less available disk space in macos-14-arm64

ehuss commented 2 weeks ago

Description

The 20240827.4 release of the macos-14-arm64 image seems to have significantly less available disk space than the previous release (20240818.4). I wanted to check if this was intended or expected. This is unfortunately causing our builds to fail.

20240827.4 initial space:

Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s1s1   295Gi   9.6Gi    18Gi    35%    404k  190M    0%   /
devfs            195Ki   195Ki     0Bi   100%     674     0  100%   /dev
/dev/disk3s6     295Gi    20Ki    18Gi     1%       0  190M    0%   /System/Volumes/VM
/dev/disk3s2     295Gi   4.9Gi    18Gi    22%     153  190M    0%   /System/Volumes/Preboot
/dev/disk3s4     295Gi   288Ki    18Gi     1%      23  190M    0%   /System/Volumes/Update
/dev/disk1s2     500Mi    20Ki   495Mi     1%       0  5.1M    0%   /System/Volumes/xarts
/dev/disk1s1     500Mi    96Ki   495Mi     1%      20  5.1M    0%   /System/Volumes/iSCPreboot
/dev/disk1s3     500Mi   196Ki   495Mi     1%      17  5.1M    0%   /System/Volumes/Hardware
/dev/disk3s5     295Gi   261Gi    18Gi    94%    2.2M  190M    1%   /System/Volumes/Data
map auto_home      0Bi     0Bi     0Bi   100%       0     0     -   /System/Volumes/Data/home

compared to 20240818.4 initial space:

Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s1s1   295Gi   9.6Gi    57Gi    15%    404k  593M    0%   /
devfs            195Ki   195Ki     0Bi   100%     674     0  100%   /dev
/dev/disk3s6     295Gi    20Ki    57Gi     1%       0  593M    0%   /System/Volumes/VM
/dev/disk3s2     295Gi   4.9Gi    57Gi     9%     153  593M    0%   /System/Volumes/Preboot
/dev/disk3s4     295Gi   272Ki    57Gi     1%      14  593M    0%   /System/Volumes/Update
/dev/disk1s2     500Mi    20Ki   495Mi     1%       0  5.1M    0%   /System/Volumes/xarts
/dev/disk1s1     500Mi    96Ki   495Mi     1%      20  5.1M    0%   /System/Volumes/iSCPreboot
/dev/disk1s3     500Mi   196Ki   495Mi     1%      17  5.1M    0%   /System/Volumes/Hardware
/dev/disk3s5     295Gi   222Gi    57Gi    80%    2.1M  593M    0%   /System/Volumes/Data
map auto_home      0Bi     0Bi     0Bi   100%       0     0     -   /System/Volumes/Data/home

Platforms affected

[ ] Azure DevOps
[X] GitHub Actions - Standard Runners
[ ] GitHub Actions - Larger Runners

Runner images affected

[ ] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] Ubuntu 24.04
[ ] macOS 12
[ ] macOS 13
[ ] macOS 13 Arm64
[ ] macOS 14
[X] macOS 14 Arm64
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link

20240827.4

https://github.com/rust-lang-ci/rust/actions/runs/10612800738/job/29415273318

Is it regression?

Yes, 20240818.4 works. https://github.com/rust-lang-ci/rust/actions/runs/10622377375/job/29446525166

Expected behavior

Would like to have more than 18GiB free when starting.

Actual behavior

Only 18GiB free when starting.

Repro steps

Run a job on macos-14-arm64 that uses the new image.

susmitamane commented 2 weeks ago

Hi @ehuss

We are looking into it. Will keep you posted.

erik-bershel commented 2 weeks ago

Hey @ehuss!

Please, see the description of GitHub hosted runners bellow:

https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories

Thank you for bringing this fact to our attention. Most likely, it is related to the fact that a new Xcode with all SDKs and simulators was added to the assembly, but this still fits into the stated service agreement and is not a regression. Please, see the description of GitHub hosted runners bellow:

https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories

Just in case, we will of course check what exactly led to such a sharp decrease in space and will try to do something about it, but unfortunately we cannot make any promises. MacOS images are the most voluminous and it is quite difficult to remove something from there without negatively affecting the experience of other users. If your builds require additional space, CPU or RAM, then I recommend paying attention to Larger Runners:

https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners/about-larger-runners

nth10sd commented 2 weeks ago

Is there any equivalent cleanup command as per #10386 but for macOS runners instead?

workingjubilee commented 2 weeks ago

I appreciate you looking into this, but regarding the recommendation to use larger runners for macOS, I am curious: which one is the one with more than 14GiB storage (as promised by service contract), exactly?

Runner Size	Architecture	Processor (CPU)	Memory (RAM)	Storage (SSD)	Workflow label
Large	Intel	12	30 GB	14 GB	macos-latest-large, macos-12-large, macos-13-large [latest], macos-14-large[Beta]
XLarge	arm64 (M1)	6 (+ 8 GPU hardware acceleration)	14 GB	14 GB	macos-latest-xlarge, macos-13-xlarge [latest], macos-14-xlarge[Beta]

erik-bershel commented 2 weeks ago

Hey @nth10sd!

We'll definitely propose something like that to you later. It's a good temporary solution I think.

erik-bershel commented 2 weeks ago

Hey @workingjubilee!

Nice input. I checked up with the corresponding team and it seams that you are right - there is no option to extend disk space for macOS as it might be done for the Ubuntu/Windows images.

Given the above and all the information in general, the best solution would be to try to increase the free space on our side - we will try to work on this. As an interim solution, we will develop a couple of small scripts that can be implemented in your CI in order to free up space on the runner by removing components that are not needed at the moment.

cc: @ehuss

the8472 commented 2 weeks ago

APFS seems to support reflinks/file clones. Has deduplication been applied to the runner images? If not then that might save some space.

igorGevaerd commented 2 weeks ago

We are suffering from the same issue too

StephenHodgson commented 1 week ago

+1 for tracking. Also running into this.

gabe4coding commented 1 week ago

Suffering of this as well +1

fabioferrero commented 1 week ago

+1 also on our side

the8472 commented 1 week ago

You can 👍 the issue instead of adding new comments.

StephenHodgson commented 1 week ago

👍

quark17 commented 1 week ago

We encountered this as a sudden failure of Homebrew to install the mactex-no-gui package, with only the cryptic message: installer: The install failed..

Searching the issues in this repo didn't turn up anything -- hopefully, mentioning Homebrew in this comment will help someone searching in the future. I was eventually able to discover where the Homebrew installer logs are recorded and saw messages that the package needed 8.89 GB but only 7.99 GB was available. The install works on macos-12 and macos-13 runners, which still have enough space. But this alerted to me how much space the mactex-no-gui bundle needed. We are able to workaround the decrease in space by installing the Homebrew texlive target instead, which is smaller.

jamescrosswell commented 1 week ago

I ran into this issue on our macos runner. After doing a bit of sleuthing with df and du to work out what was consuming so much disk space I discovered all of these in the Applications folder:

44G /Applications
 11G    /Applications/Xcode_14.3.1.app
 4.9G   /Applications/Xcode_15.4.app
 4.9G   /Applications/Xcode_15.3.app
 4.8G   /Applications/Xcode_15.2.app
 4.5G   /Applications/Xcode_15.1.app
 4.5G   /Applications/Xcode_15.0.1.app
 4.3G   /Applications/Xcode_16_beta_6.app
 4.3G   /Applications/Xcode_16.1_beta.app

We do use Xcode as part of our build, but we only need one of those versions of Xcode (not all of them). So the solution in our case was to delete a bunch of the old ones:

      # We only use Xcode 15.4
      - name: Remove unused applications
        run: |
          df -hI /dev/disk3s1s1
          sudo rm -rf /Applications/Xcode_14.3.1.app
          sudo rm -rf /Applications/Xcode_15.0.1.app
          sudo rm -rf /Applications/Xcode_15.1.app
          sudo rm -rf /Applications/Xcode_15.2.app
          sudo rm -rf /Applications/Xcode_15.3.app
          df -hI /dev/disk3s1s1

... which frees up about 30GB of space.

Galkon commented 1 week ago

Running into non-stop issues tonight on macos-14 runners, I suspect related to this. It is during creation + publishing of pkg and dmg files for an electron app using electron-builder.

Confirming the above solves it (delete xcode).

    '> Highlight@0.0.118-alpha clean:electron\n' +
    '> rimraf release\n' +
    '\n' +
    '  • electron-builder  version=24.13.3 os=23.6.0\n' +
    '  • loaded configuration  file=package.json ("build" field)\n' +
    '  • skipped dependencies rebuild  reason=npmRebuild is set to false\n' +
    '  • packaging       platform=darwin arch=arm64 electron=31.4.0 appOutDir=release/mac-arm64\n' +
    '  • downloading     url=https://github.com/electron/electron/releases/download/v31.4.0/electron-v31.4.0-darwin-arm64.zip size=96 MB parts=6\n' +
    '  • downloaded      url=https://github.com/electron/electron/releases/download/v31.4.0/electron-v31.4.0-darwin-arm64.zip duration=2.38s\n' +
    '  • signing         file=release/mac-arm64/Highlight.app platform=darwin type=distribution identity=... provisioningProfile=...\n' +
    '  • notarization successful\n' +
    '  • building        target=macOS zip arch=arm64 file=release/Highlight-0.0.118-alpha-arm64-mac.zip\n' +
    '  • building        target=DMG arch=arm64 file=release/Highlight-0.0.118-alpha-arm64.dmg\n' +
    '  • building        target=pkg arch=arm64 file=release/Highlight-0.0.118-alpha-arm64.pkg\n' +
    '  • Detected arm64 process, HFS+ is unavailable. Creating dmg with APFS - supports Mac OSX 10.12+\n' +
    '  • building block map  blockMapFile=release/Highlight-0.0.118-alpha-arm64.dmg.blockmap\n' +
    '  • publishing      publisher=[object Object]\n' +
    'Preparing for release: /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/Highlight-0.0.118-alpha-arm64.dmg.blockmap\n' +
    'Preparing for release: /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/Highlight-0.0.118-alpha-arm64.dmg\n' +
    '  • building block map  blockMapFile=release/Highlight-0.0.118-alpha-arm64-mac.zip.blockmap\n' +
    'Preparing for release: /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/Highlight-0.0.118-alpha-arm64-mac.zip.blockmap\n' +
    'Preparing for release: /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/Highlight-0.0.118-alpha-arm64-mac.zip\n' +
    '  ⨯ Exit code: 1. Command failed: productbuild --synthesize --component /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/mac-arm64/Highlight.app /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/distribution.xml\n' +
    `productbuild: error: Can't write temporary package to "/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/TemporaryItems/NSIRD_productbuild_KbOv6A/tv.medal.highlight.pkg" (Cannot write package to "/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/TemporaryItems/NSIRD_productbuild_KbOv6A/tv.medal.highlight.pkg".).\n` +
    '\n' +
    '\n' +
    'productbuild: Adding component at /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/mac-arm64/Highlight.app\n' +
    'productbuild: Inferred install-location of /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/mac-arm64\n' +
    '\n' +
    `productbuild: error: Can't write temporary package to "/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/TemporaryItems/NSIRD_productbuild_KbOv6A/tv.medal.highlight.pkg" (Cannot write package to "/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/TemporaryItems/NSIRD_productbuild_KbOv6A/tv.medal.highlight.pkg".).\n` +
    '\n' +
    '  failedTask=build stackTrace=Error: Exit code: 1. Command failed: productbuild --synthesize --component /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/mac-arm64/Highlight.app /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/distribution.xml\n' +
    `productbuild: error: Can't write temporary package to "/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/TemporaryItems/NSIRD_productbuild_KbOv6A/tv.medal.highlight.pkg" (Cannot write package to "/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/TemporaryItems/NSIRD_productbuild_KbOv6A/tv.medal.highlight.pkg".).\n` +
    '                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        productbuild: Adding component at /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/mac-arm64/Highlight.app\n' +
    'productbuild: Inferred install-location of /private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/release/mac-arm64\n' +
    '                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             '... 2317 more characters,
  stderr: '\x1B[1m\x1B[33m[plugin:vite:resolve]\x1B[39m\x1B[22m \x1B[33m[plugin vite:resolve] Module "crypto" has been externalized for browser compatibility, imported by "/private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/src/main/utils/crypto/JsonCrypto.ts". See https://vitejs.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility for more details.\x1B[39m\n' +
    '\x1B[1m\x1B[33m[plugin:vite:resolve]\x1B[39m\x1B[22m \x1B[33m[plugin vite:resolve] Module "fs" has been externalized for browser compatibility, imported by "/private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/src/shared/utils/StringUtils.ts". See https://vitejs.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility for more details.\x1B[39m\n' +
    '\x1B[1m\x1B[33m[plugin:vite:resolve]\x1B[39m\x1B[22m \x1B[33m[plugin vite:resolve] Module "util" has been externalized for browser compatibility, imported by "/private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/node_modules/handlebars-utils/index.js". See https://vitejs.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility for more details.\x1B[39m\n' +
    '\x1B[1m\x1B[33m[plugin:vite:resolve]\x1B[39m\x1B[22m \x1B[33m[plugin vite:resolve] Module "stream" has been externalized for browser compatibility, imported by "/private/var/folders/m4/5dz5h26x329cqq4fx333f8gm0000gn/T/highlight-electron-temp/standalone/node_modules/handlebars-async-helpers/helpers/each.js". See https://vitejs.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility for more details.\x1B[39m\n' +
    'node_modules/mammoth/node_modules/bluebird/js/release/util.js (201:9): Use of eval in "node_modules/mammoth/node_modules/bluebird/js/release/util.js" is strongly discouraged as it poses security risks and may cause issues with minification.\n'

Vyazovoy commented 1 week ago

Here is a step that will delete all beta versions of Xcode unless you explicitly selected one:

- name: Delete unused Beta Xcode installations
   run: |
     echo "Available Xcode installations:"
     xcodes installed
     selected_path="$(xcode-select -p)"; selected_path="${selected_path%/Contents/Developer}"
     find /Applications \
       -depth 1 \
       -name "Xcode*beta*.app" \
       -not -path "${selected_path}" \
       -exec rm -rf {} +

JoeSzymanskiFAN commented 1 week ago

In addition to the above suggestions to delete unused versions of Xcode, we were able to add a significant amount of space by deleting unused simulators and clearing the simulator cache:

echo "Deleting all iOS simulators"
xcrun simctl delete all

# Do the following only if you need a simulator for things like unit tests
echo "Creating an iOS simulator to run unit tests"
xcrun simctl create "Unit-Test-Device" "iPhone 15 Pro Max" "iOS17.5"

# This folder starts at 42 GB and sometimes increases to as high as 61 GB.
# After deleting it and running a build, the final size is only 12 GB, so this is
# a good target for disk space savings.
echo "Deleting iOS Simulator caches"
sudo rm -rf ~/Library/Developer/CoreSimulator/Caches/*

yasali commented 1 week ago

I have tried below script on my unit test job, and after cleaning up the space and when I rerun the job, all Xcodes version are still there.

- name: Calculate initial disk space
  run: df -h

- name: Size of apps in Applications folder Before deletion
  run: |
    sudo du -sh /Applications/*

# We only use Xcode 15.4
- name: Remove unused applications
  run: |
    df -hI /dev/disk3s1s1
    sudo rm -rf /Applications/Xcode_14.3.1.app
    sudo rm -rf /Applications/Xcode_15.0.1.app
    sudo rm -rf /Applications/Xcode_15.1.app
    sudo rm -rf /Applications/Xcode_15.2.app
    sudo rm -rf /Applications/Xcode_15.3.app
    df -hI /dev/disk3s1s1

- name: Size of apps in Applications folder After deletion
  run: |
    sudo du -sh /Applications/*

- name: Clean Xcode derived data and caches
  run: |
    sudo rm -rf ~/Library/Developer/Xcode/DerivedData

- name: Delete unnecessary simulator files
  run: |
    sudo rm -rf ~/Library/Developer/CoreSimulator/Devices/*
    sudo rm -rf ~/Library/Developer/CoreSimulator/Caches/*

- name: Clean up iOS DeviceSupport files
  run: |
    sudo rm -rf ~/Library/Developer/Xcode/iOS\ DeviceSupport/*

- name: Check final disk space
  run: df -h

chandlerc commented 4 days ago

For other folks finding this...

Definitely bit our project too, and seems like a pretty sharp regression.

We found removing Xcode versions was very slow though and for limited space. The tip to remove the iOS simulators (and maybe only add the one you need) look much more promising. Happens that we're just testing on macOS, not iOS, so even better for us. In case folks want to see a full PR that adds this: https://github.com/carbon-language/carbon-lang/pull/4287

rodperottoni commented 4 days ago

Also ran into this issue. My actions are constantly failing unless I do some serious runner cleanup beforehand. Upon investigation, it seems like every Xcode folder in Applications has a duplicate with an extraneous _0 at the end.

Running: find /Applications -name "Xcode_*" -maxdepth 1 -mindepth 1

Outputs:

/Applications/Xcode_16.0.app
/Applications/Xcode_16.1.app
/Applications/Xcode_14.3.app
/Applications/Xcode_15.1.app
/Applications/Xcode_15.0.app
/Applications/Xcode_15.2.app
/Applications/Xcode_15.3.app
/Applications/Xcode_15.4.app
/Applications/Xcode_14.3.1.app
/Applications/Xcode_16.1_beta.app
/Applications/Xcode_15.1.0.app
/Applications/Xcode_16.0.0.app
/Applications/Xcode_15.3.0.app
/Applications/Xcode_15.4.0.app
/Applications/Xcode_15.2.0.app
/Applications/Xcode_16.1.0.app
/Applications/Xcode_16_beta_6.app
/Applications/Xcode_15.0.1.app

Is this expected? Why suddenly all versions have a sibling .0 clone?

erik-bershel commented 4 days ago

Hey @rodperottoni!

It's just a symlink. It takes no space.

actions / runner-images