search

actions / runner-images

GitHub Actions runner images
MIT License
10.07k stars 3.04k forks source link

Miserable reliability with macos-14 runners #10680

Open Galkon opened 3 weeks ago

Galkon commented 3 weeks ago

Description

Over the last ~1 month, the macos-14 runners reliability has seriously degraded. We have used them since March, and used to have relatively no issues. We have absolutely no issues with our self-hosted macOS intel runner, so I am about to do the same for apple silicon.

At this point they are borderline unusable, having to manually delete xcode and iOS simulators just for the drive to have free space is a bit insane. And even with that, random failures that are not addressable by code or config changes frequently occur.

Platforms affected

Runner images affected

Image version and build link

Image: macos-14-arm64 Version: 20240918.8 Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/20240[9](https://github.com/highlight-ing/highlight/actions/runs/11022480898/job/30613014291#step:1:10)18.8/images/macos/macos-14-arm64-Readme.md Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20240918.8

Not public, but here is a link: https://github.com/highlight-ing/highlight/actions/runs/11022480898/job/30613014291

Is it regression?

Yes. https://github.com/highlight-ing/highlight/actions/runs/11021711293/job/30609399929 Image: macos-14-arm64 Version: 20240918.8 Included Software: https://github.com/actions/runner-images/blob/macos-14-arm64/20240918.8/images/macos/macos-14-arm64-Readme.md Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20240918.8

Expected behavior

Builds should reliably work and not randomly throw errors that go away when you re-run them half a dozen times.

Actual behavior

I play russian roulette with github actions to see if my builds work or if I need to spend 4 hours re-running until it does.

Repro steps

No reliable reproduction. It just happens sometimes (regularly) while building an electron app for macOS arm64 using electron-builder.

erik-bershel commented 3 weeks ago

Hey @Galkon!

Latest image version runners will provide more precious space to run your builds. And, if I understand correctly how chunks rendering works, it should resolve your issue. I don't see a clear reason for the crash in this log, but most likely the process didn't have enough disk space or other resources (which is less likely) to complete. Considering that the project is not public, I will not be able to get enough information for a quality investigation - we do not have access to private runs. I suggest waiting for the end of the release and trying your build on the new runner. Will you be able to conduct a couple of test launches on Thursday or Friday (deployment should take not less than 15 hours) and share the results with us?

Galkon commented 3 weeks ago

Hey @Galkon!

Latest image version runners will provide more precious space to run your builds. And, if I understand correctly how chunks rendering works, it should resolve your issue. I don't see a clear reason for the crash in this log, but most likely the process didn't have enough disk space or other resources (which is less likely) to complete. Considering that the project is not public, I will not be able to get enough information for a quality investigation - we do not have access to private runs. I suggest waiting for the end of the release and trying your build on the new runner. Will you be able to conduct a couple of test launches on Thursday or Friday (deployment should take not less than 15 hours) and share the results with us?

Thanks for the swift response! Happy to be your guinea pig. Thursday and Friday we are typically doing releases anyways.

nsheaps commented 2 weeks ago

I know adding a +1 to an issue is typically not a great practice but this also seriously impacts our private builds.

I had filed a ticket with github support: https://support.github.com/ticket/personal/0/2996091 The response gave no indication as to why the reliability had changed, or what changes were made that might have caused it.

This had affected one of our 4 macos workflows, now affects 2, to the point that they never pass.

As part of our debugging, we confirmed that macos-latest-large had the issue, and hadn't changed to apple silicon nor had the OS build changed between previous successful one and now.

When they do fail, there's 2 possible outcomes:

  1. The run is marked as cancelled, and the workflow logs miss the logs from the last step, even though you can see them streaming at run time
  2. The run is makred as skipped, even though the run does attempt to run

In both cases, the workflow time breakdown for the macos runner always caps out just over 30m, even though the step execution time (if available) doesn't align with that. The "instability" and attention by support has gotten to the point of considering a different vendor for mac-os based CI builds.

If possible, please also include me on the test runs on thursday/friday. Happy to grant support access to our private repo as well for help debugging, or hop on a call to debug further.

erik-bershel commented 2 weeks ago

Hey @Galkon!

macOS-14 images without visionOS on board rolled out. It made free plenty of precious GB.

Please check on the workflows mentioned state and come back to us with the updates. 🙇

erik-bershel commented 2 weeks ago

Hey @nsheaps!

I'm not sure that your problem and the original problem stated in the issue are related. It's hard to deny such a possibility without logs, examples and a test project, yes. But the description doesn't really match up. If you think that the problem was related to disk space, then the last completed update should have solved it for you. But based on the description, I can say that it looks more like your workflow has exhausted the runner's resources. I can't say for sure without a workflow example and links to successful and unsuccessful launches - then I can at least track the version of the image used and the connection between the updates of the software used and the crashes.

Galkon commented 2 weeks ago

Hey @Galkon!

macOS-14 images without visionOS on board rolled out. It made free plenty of precious GB.

Please check on the workflows mentioned state and come back to us with the updates. 🙇

Thanks for the update. We will be running several builds throughout today and tomorrow, I will let you know if I run into any issues.

nsheaps commented 2 weeks ago

I think in some way it is related, as we had stable performance prior, then some change was made and then our workflows started failing in weird ways.

But based on the description, I can say that it looks more like your workflow has exhausted the runner's resources.

We did note that disk space usage (measured by du and df) went above the documented thresholds, but if that were to be the case, we built a known working commit prior and it still failed, potentially due to the visionOS changes in the macos image. We'll give our workflows a shot again with macos-latest-large and macos-14-large and see if they've recovered after this change.

I think it's also somewhat important to note that this affected multiple workflows, not just the ones with high disk usage, so we'll give those a shot too. We've also documented internally to look for other mechanisms as the disk usage is coming from running our entire stack on the runner itself so we can run e2e tests with playwright, which isn't a lightweight task, but unfortunately that move is not a quick one to make.

I'll report back later today with the results of our testing. If it ends up not working still, I'm happy to toss any info your way either here or in a new ticket.

Appreciate the updates and attention here @erik-bershel !

erik-bershel commented 2 weeks ago

@nsheaps 👋 Just to be on the same page. The only notable difference from productivity perspective between macos-14 and macos-14-xlarge is threads and RAM amount. And there is no difference between macos-latest-large and macos-14-large - it's the same image, just different labels/tags, the same is applicable for macos-latest and macos-14. The difference between macos-14(latest)-xlarge and macos-14(latest)-large might be found here: https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners/about-larger-runners#about-macos-larger-runners If your problems were specifically related to disk space (before the current release, this problem was common to all macOS-14 based images, especially arm64 based macOS-14). As of today, we have freed up a huge amount of space, which should have eliminated this problem. I expect that this should solve the problem that was originally described in the issue - chunk baking is a procedure that is quite sensitive to disk space. At the same time, I am not aware of any other major issues that have been discovered recently that could lead to blocking your workflows. Accordingly, some other incompatibility or problem may be discovered here (perhaps your builds block the runner due to other resources or the number of disk requests, which leads to the machine hanging and the connection being broken, which could have been caused by updates to packages that participate in the build). To investigate such cases, we always need a lot of information. So far, it is not enough even to make assumptions.

Galkon commented 2 weeks ago

@erik-bershel it looks like we are still having issues.

image

The issue is the chunking issue mentioned in prior posts.

    '\x1B[36mvite v5.4.8 \x1B[32mbuilding SSR bundle for production...\x1B[36m\x1B[39m\n' +
    'transforming...\n' +
    '\x1B[32m✓\x1B[39m 294 modules transformed.\n' +
    'rendering chunks...\n' +
    '\x1B[2mout/main/\x1B[22m\x1B[2mchunks/\x1B[22m\x1B[36mHashUtils.renderer-hSXnO4Rp.js  \x1B[39m\x1B[1m\x1B[2m  0.36 kB\x1B[22m\x1B[1m\x1B[22m\x1B[2m │ map:     0.89 kB\x1B[22m\n' +
    '\x1B[2mout/main/\x1B[22m\x1B[2mchunks/\x1B[22m\x1B[36mwriteAtomicFile-CEaw96OJ.js     \x1B[39m\x1B[1m\x1B[2m  2.19 kB\x1B[22m\x1B[1m\x1B[22m\x1B[2m │ map:    10.04 kB\x1B[22m\n' +
    '\x1B[2mout/main/\x1B[22m\x1B[2mchunks/\x1B[22m\x1B[36mJsonCrypto-ByPgB7sW.js          \x1B[39m\x1B[1m\x1B[2m  2.56 kB\x1B[22m\x1B[1m\x1B[22m\x1B[2m │ map:     4.11 kB\x1B[22m\n' +
    '\x1B[2mout/main/\x1B[22m\x1B[36mindex.js                               \x1B[39m\x1B[1m\x1B[2m475.80 kB\x1B[22m\x1B[1m\x1B[22m\x1B[2m │ map: 1,607.65 kB\x1B[22m\n' +
    '\x1B[32m✓ built in 3.47s\x1B[39m\n' +
    '\x1B[36mvite v5.4.8 \x1B[32mbuilding SSR bundle for production...\x1B[36m\x1B[39m\n' +
    'transforming...\n' +
    '\x1B[32m✓\x1B[39m 11 modules transformed.\n' +
    'rendering chunks...\n' +
    '\x1B[2mout/preload/\x1B[22m\x1B[36minternal.js     \x1B[39m\x1B[1m\x1B[2m 0.71 kB\x1B[22m\x1B[1m\x1B[22m\x1B[2m │ map:  2.47 kB\x1B[22m\n' +
    '\x1B[2mout/preload/\x1B[22m\x1B[36melectron.js     \x1B[39m\x1B[1m\x1B[2m 2.37 kB\x1B[22m\x1B[1m\x1B[22m\x1B[2m │ map:  8.86 kB\x1B[22m\n' +
    '\x1B[2mout/preload/\x1B[22m\x1B[36mapp-runtime.js  \x1B[39m\x1B[1m\x1B[2m17.78 kB\x1B[22m\x1B[1m\x1B[22m\x1B[2m │ map: 57.81 kB\x1B[22m\n' +
    '\x1B[32m✓ built in 330ms\x1B[39m\n' +
    '\x1B[36mvite v5.4.8 \x1B[32mbuilding for production...\x1B[36m\x1B[39m\n' +
    'transforming...\n' +
    '\x1B[32m✓\x1B[39m 4773 modules transformed.\n' +
    'rendering chunks...\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[32msplash.html                                      \x1B[39m\x1B[1m\x1B[2m    1.67 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[32mfullscreen-overlay.html                          \x1B[39m\x1B[1m\x1B[2m    2.33 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[32mbase.html                                        \x1B[39m\x1B[1m\x1B[2m    3.38 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Size3-Regular-CTq5MqoE.woff         \x1B[39m\x1B[1m\x1B[2m    4.42 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Size4-Regular-Dl5lxZxV.woff2        \x1B[39m\x1B[1m\x1B[2m    4.93 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Size2-Regular-Dy4dx90m.woff2        \x1B[39m\x1B[1m\x1B[2m    5.21 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Size1-Regular-mCD8mA8B.woff2        \x1B[39m\x1B[1m\x1B[2m    5.47 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Size4-Regular-BF-4gkZK.woff         \x1B[39m\x1B[1m\x1B[2m    5.98 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Size2-Regular-oD1tc_U0.woff         \x1B[39m\x1B[1m\x1B[2m    6.19 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Size1-Regular-C195tn64.woff         \x1B[39m\x1B[1m\x1B[2m    6.50 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Caligraphic-Regular-Di6jR-x-.woff2  \x1B[39m\x1B[1m\x1B[2m    6.91 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Caligraphic-Bold-Dq_IR9rO.woff2     \x1B[39m\x1B[1m\x1B[2m    6.91 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Size3-Regular-DgpXs0kz.ttf          \x1B[39m\x1B[1m\x1B[2m    7.59 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Caligraphic-Regular-CTRA-rTL.woff   \x1B[39m\x1B[1m\x1B[2m    7.66 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Caligraphic-Bold-BEiXGLvX.woff      \x1B[39m\x1B[1m\x1B[2m    7.72 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Script-Regular-D3wIWfF6.woff2       \x1B[39m\x1B[1m\x1B[2m    9.64 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_SansSerif-Regular-DDBCnlJ7.woff2    \x1B[39m\x1B[1m\x1B[2m   10.34 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Size4-Regular-DWFBv043.ttf          \x1B[39m\x1B[1m\x1B[2m   10.36 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Script-Regular-D5yQViql.woff        \x1B[39m\x1B[1m\x1B[2m   10.59 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Fraktur-Regular-CTYiF6lA.woff2      \x1B[39m\x1B[1m\x1B[2m   11.32 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Fraktur-Bold-CL6g_b3V.woff2         \x1B[39m\x1B[1m\x1B[2m   11.35 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Size2-Regular-B7gKUWhC.ttf          \x1B[39m\x1B[1m\x1B[2m   11.51 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_SansSerif-Italic-C3H0VqGB.woff2     \x1B[39m\x1B[1m\x1B[2m   12.03 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_SansSerif-Bold-D1sUS0GD.woff2       \x1B[39m\x1B[1m\x1B[2m   12.22 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Size1-Regular-Dbsnue_I.ttf          \x1B[39m\x1B[1m\x1B[2m   12.23 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_SansSerif-Regular-CS6fqUqJ.woff     \x1B[39m\x1B[1m\x1B[2m   12.32 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Caligraphic-Regular-wX97UBjC.ttf    \x1B[39m\x1B[1m\x1B[2m   12.34 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Caligraphic-Bold-ATXxdsX0.ttf       \x1B[39m\x1B[1m\x1B[2m   12.37 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Fraktur-Regular-Dxdc4cR9.woff       \x1B[39m\x1B[1m\x1B[2m   13.21 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Fraktur-Bold-BsDP51OF.woff          \x1B[39m\x1B[1m\x1B[2m   13.30 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Typewriter-Regular-CO6r4hn1.woff2   \x1B[39m\x1B[1m\x1B[2m   13.57 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_SansSerif-Italic-DN2j7dab.woff      \x1B[39m\x1B[1m\x1B[2m   14.11 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_SansSerif-Bold-DbIhKOiC.woff        \x1B[39m\x1B[1m\x1B[2m   14.41 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Typewriter-Regular-C0xS9mPB.woff    \x1B[39m\x1B[1m\x1B[2m   16.03 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Math-BoldItalic-CZnvNsCZ.woff2      \x1B[39m\x1B[1m\x1B[2m   16.40 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Math-Italic-t53AETM-.woff2          \x1B[39m\x1B[1m\x1B[2m   16.44 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Script-Regular-C5JkGWo-.ttf         \x1B[39m\x1B[1m\x1B[2m   16.65 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-BoldItalic-DxDJ3AOS.woff2      \x1B[39m\x1B[1m\x1B[2m   16.78 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-Italic-NWA7e6Wa.woff2          \x1B[39m\x1B[1m\x1B[2m   16.99 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Math-BoldItalic-iY-2wyZ7.woff       \x1B[39m\x1B[1m\x1B[2m   18.67 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Math-Italic-DA0__PXp.woff           \x1B[39m\x1B[1m\x1B[2m   18.75 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-BoldItalic-SpSLRI95.woff       \x1B[39m\x1B[1m\x1B[2m   19.41 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_SansSerif-Regular-BNo7hRIc.ttf      \x1B[39m\x1B[1m\x1B[2m   19.44 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Fraktur-Regular-CB_wures.ttf        \x1B[39m\x1B[1m\x1B[2m   19.57 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Fraktur-Bold-BdnERNNW.ttf           \x1B[39m\x1B[1m\x1B[2m   19.58 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-Italic-BMLOBm91.woff           \x1B[39m\x1B[1m\x1B[2m   19.68 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_SansSerif-Italic-YYjJ1zSn.ttf       \x1B[39m\x1B[1m\x1B[2m   22.36 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_SansSerif-Bold-CFMepnvq.ttf         \x1B[39m\x1B[1m\x1B[2m   24.50 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-Bold-Cx986IdX.woff2            \x1B[39m\x1B[1m\x1B[2m   25.32 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-Regular-B22Nviop.woff2         \x1B[39m\x1B[1m\x1B[2m   26.27 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Typewriter-Regular-D3Ib7_Hf.ttf     \x1B[39m\x1B[1m\x1B[2m   27.56 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_AMS-Regular-BQhdFMY1.woff2          \x1B[39m\x1B[1m\x1B[2m   28.08 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-Bold-Jm3AIy58.woff             \x1B[39m\x1B[1m\x1B[2m   29.91 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-Regular-Dr94JaBh.woff          \x1B[39m\x1B[1m\x1B[2m   30.77 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Math-BoldItalic-B3XSjfu4.ttf        \x1B[39m\x1B[1m\x1B[2m   31.20 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Math-Italic-flOr_0UB.ttf            \x1B[39m\x1B[1m\x1B[2m   31.31 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-BoldItalic-DzxPMmG6.ttf        \x1B[39m\x1B[1m\x1B[2m   32.97 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_AMS-Regular-DMm9YOAa.woff           \x1B[39m\x1B[1m\x1B[2m   33.52 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-Italic-3WenGoN9.ttf            \x1B[39m\x1B[1m\x1B[2m   33.58 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-Bold-waoOVXN0.ttf              \x1B[39m\x1B[1m\x1B[2m   51.34 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_Main-Regular-ypZvNtVU.ttf           \x1B[39m\x1B[1m\x1B[2m   53.58 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mFullDiskAccess-B5uxDzHu.png               \x1B[39m\x1B[1m\x1B[2m   56.45 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[32mKaTeX_AMS-Regular-DRggAlZN.ttf            \x1B[39m\x1B[1m\x1B[2m   63.63 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[35mfullscreen-overlay-BilsSMWy.css           \x1B[39m\x1B[1m\x1B[2m   28.45 kB\x1B[22m\x1B[1m\x1B[22m\n' +
    '\x1B[2m../../out/renderer/\x1B[22m\x1B[2massets/\x1B[22m\x1B[36m__vite-browser-external-BjGS5_Dj.js       \x1B[39m\x1B[1m\x1B[2m    0.22 kB\x1B[22m\x1B[1m\x1B[22m\x1B[2m │ map: '... 5033 more characters,
  stderr: '\x1B[1m\x1B[33m[plugin:vite:resolve]\x1B[39m\x1B[22m \x1B[33m[plugin vite:resolve] Module "crypto" has been externalized for browser compatibility, imported by "/private/var/folders/0g/hj_q_pzx65bbjnslxz9n0src0000gn/T/highlight-electron-temp/standalone/src/shared/ipc/ProcessWideEvents.ts". See https://vitejs.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility for more details.\x1B[39m\n' +
    '\x1B[1m\x1B[33m[plugin:vite:resolve]\x1B[39m\x1B[22m \x1B[33m[plugin vite:resolve] Module "util" has been externalized for browser compatibility, imported by "/private/var/folders/0g/hj_q_pzx65bbjnslxz9n0src0000gn/T/highlight-electron-temp/standalone/node_modules/handlebars-utils/index.js". See https://vitejs.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility for more details.\x1B[39m\n' +
    "node_modules/framer-motion/dist/es/components/AnimatePresence/index.mjs (1:0): Error when using sourcemap for reporting an error: Can't resolve original location of error.\n" +
    "node_modules/framer-motion/dist/es/motion/index.mjs (1:0): Error when using sourcemap for reporting an error: Can't resolve original location of error.\n" +
    '\x1B[1m\x1B[33m[plugin:vite:resolve]\x1B[39m\x1B[22m \x1B[33m[plugin vite:resolve] Module "stream" has been externalized for browser compatibility, imported by "/private/var/folders/0g/hj_q_pzx65bbjnslxz9n0src0000gn/T/highlight-electron-temp/standalone/node_modules/handlebars-async-helpers/helpers/each.js". See https://vitejs.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility for more details.\x1B[39m\n' +
    '\x1B[1m\x1B[33m[plugin:vite:resolve]\x1B[39m\x1B[22m \x1B[33m[plugin vite:resolve] Module "crypto" has been externalized for browser compatibility, imported by "/private/var/folders/0g/hj_q_pzx65bbjnslxz9n0src0000gn/T/highlight-electron-temp/standalone/src/main/utils/crypto/JsonCrypto.ts". See https://vitejs.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility for more details.\x1B[39m\n' +
    '\x1B[1m\x1B[33m[plugin:vite:resolve]\x1B[39m\x1B[22m \x1B[33m[plugin vite:resolve] Module "fs" has been externalized for browser compatibility, imported by "/private/var/folders/0g/hj_q_pzx65bbjnslxz9n0src0000gn/T/highlight-electron-temp/standalone/src/shared/utils/StringUtils.ts". See https://vitejs.dev/guide/troubleshooting.html#module-externalized-for-browser-compatibility for more details.\x1B[39m\n' +
    'node_modules/mammoth/node_modules/bluebird/js/release/util.js (201:9): Use of eval in "node_modules/mammoth/node_modules/bluebird/js/release/util.js" is strongly discouraged as it poses security risks and may cause issues with minification.\n'
}
erik-bershel commented 2 weeks ago

Hey @Galkon! Sorry to hear it. So, that wasn't disk originally. 😞 Then we will need more information for investigation. Can you provide more complete logs? Probably further we will need an example of a project where instability of this kind is observed or at least more recognisable steps of reproduction for localisation of the problem. If suddenly you can find in the history of launches starting from which version of the image this problem began, that would also be very useful.

nsheaps commented 2 weeks ago

@erik-bershel looks like some of our failures on one workflow might have been related to our code but one of our workflows (an e2e test workflow) still is not working with macos-14(latest)-large. I still think this might be a disk issue, though it seems like @Galkon's issue is not that. Happy to open a new ticket to investigate further too but as one final check, I'm kicking off that workflow with macos-13-large. Happy to hop on a call/grant support access to debug further or share any needed logs

Based on what you shared it sounds like perhaps the visionOS stuff was included in all available macos images but was only removed from macos14 so not sure if it will work, but I know a past commit run with macos-latest-large on june 19th succeeded (unfortunately logs are gone so I don't know which OS version etc).

On a known good commit, we added some debug info too, before and after snapshots of df are below (regenerating in human readable, will edit in a moment). Based on this, the device has a lot of free space so it's not the OS killing it, but perhaps the "14GB" limit in the host needs to be updated for when it's killed based on tools that have been added or removed over the past few months?

# just after checkout
df -ah: 
Filesystem       Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk1s5s1  325Gi  8.7Gi  172Gi     5%  356840 1807777200    0%   /
devfs           183Ki  183Ki    0Bi   100%     634          0  100%   /dev
/dev/disk1s2    325Gi  1.9Gi  172Gi     2%     906 1807777200    0%   /System/Volumes/Preboot
/dev/disk1s4    325Gi  1.0Mi  172Gi     1%       1 1807777200    0%   /System/Volumes/VM
/dev/disk1s6    325Gi  448Ki  172Gi     1%      20 1807777200    0%   /System/Volumes/Update
/dev/disk1s1    325Gi  140Gi  172Gi    45% 28[11](https://github.com/gathertown/gather-town/actions/runs/11077741486/job/30783827078#step:3:12)957 1807777200    0%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home

# just before running localstack (after installing docker, some tools, node, node dependencies, python, clean database + migrations and seeds, and redis
df -ah: 
Filesystem       Size   Used  Avail Capacity iused      ifree %iused  Mounted on
/dev/disk1s5s1  325Gi  8.7Gi  151Gi     6%  356840 1578898520    0%   /
devfs           183Ki  183Ki    0Bi   100%     634          0  100%   /dev
/dev/disk1s2    325Gi  1.9Gi  151Gi     2%     906 1578898520    0%   /System/Volumes/Preboot
/dev/disk1s4    325Gi  1.0Mi  151Gi     1%       1 1578898520    0%   /System/Volumes/VM
/dev/disk1s6    325Gi  392Ki  151Gi     1%      20 1578898520    0%   /System/Volumes/Update
/dev/disk1s1    325Gi  [16](https://github.com/gathertown/gather-town/actions/runs/11077741486/job/30783827078#step:19:17)2Gi  151Gi    52% 3817465 1578898520    0%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0          0  100%   /System/Volumes/Data/home

Let me know what other info I can provide here

EDIT: After noting the disk usage, there is definitely disk exhaustion. Docs say you have 14GB available and this shows 20 being used, though worth noting that space consumed can definitely also include space used by the runner and logs. What I don't understand is why the same commit could have such a drastically different performance (From working to absolutely broken), with the one possible exception of localstack being localstack and how they recently changed the container to download all the dependencies at runtime.

Now that I mention that, I just took a look and our dockerfile for localstack doesn't lock the base image, gonna try locking it to something lower and see if that helps.