search

phetsims / aqua

Automatic QUality Assurance
MIT License
2 stars 4 forks source link

Increase the size of `/tmp` in sparky #172

Closed zepumph closed 1 year ago

zepumph commented 1 year ago

From https://github.com/phetsims/special-ops/issues/234 It is currently only a few MB. I think that 10GB is a good starting point.

We are encountering build failures because of this:

circuit-construction-kit-dc : build
Build failed with status code 1:
Running "report-media" task

Running "clean" task

Running "build" task
>> tsc complete: 49361ms
Building runnable repository (circuit-construction-kit-dc, brands: phet, phet-io)
Building brand: phet
>> Webpack build complete: 10422ms
>> Production minification complete: 47301ms (2639646 bytes)
>> Debug minification complete: 2ms (25054172 bytes)
Building brand: phet-io
>> Webpack build complete: 9756ms
>> Production minification complete: 40901ms (2654099 bytes)
>> Debug minification complete: 38220ms (2961856 bytes)
No migration rules found at ../chipper/dist/js/phet-io-sim-specific/repos/circuit-construction-kit-dc/js/circuit-construction-kit-dc-migration-rules.js, no rules to be bundled with phet-io.js.
>> tsc complete: 6815ms
>> tsc complete: 7881ms
Fatal error: Perennial task failed:
Error: Failed to launch the browser process!
[0428/100136.995278:WARNING:discardable_shared_memory_manager.cc(197)] Less than 64MB of free space in temporary directory for shared memory files: 0

TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md

at onClose (/data/share/phet/continuous-testing/ct-main/ct-snapshots/1682694972361/chipper/node_modules/puppeteer-core/lib/cjs/puppeteer/node/BrowserRunner.js:299:20)
at ChildProcess.<anonymous> (/data/share/phet/continuous-testing/ct-main/ct-snapshots/1682694972361/chipper/node_modules/puppeteer-core/lib/cjs/puppeteer/node/BrowserRunner.js:290:24)
at ChildProcess.emit (node:events:525:35)
at ChildProcess._handle.onexit (node:internal/child_process:291:12)
Full Error details:
Error: Failed to launch the browser process!
[0428/100136.995278:WARNING:discardable_shared_memory_manager.cc(197)] Less than 64MB of free space in temporary directory for shared memory files: 0

TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md

Snapshot from 4/28/2023, 9:16:12 AM

Over in https://github.com/phetsims/aqua/issues/145 we decided to also move the temp dir for all CT browser clients, and so we could do that for the build puppeteer instances too as a second option if thing are still failing after increasing /tmp to a "normal" size.

@mattpen please expand /tmp on sparky to 10GB.

mattpen commented 1 year ago

I ran sudo mount -o size=10G,rw,nodev,nosuid -t tmpfs tmpfs /tmp which accomplished the goal of exanding /tmp to 10G. This deleted any data that was in place in /tmp. I do not believe that this will persist a reboot. I poked around and saw that phet-server2 does not have a dedicated fs for /tmp. This could be problematic if something decides to save a bunch of temporary files, but it doesn't seem like there are any problems at the moment. phet-server however, does have a tmpfs dedicated to /tmp (at a whopping 126G!). But, this tmpfs is not saved in /etc/fstab so I'm stumped how it is getting remounted after reboots.

mattpen commented 1 year ago

I tested this out with a reboot. As expected, the tmpfs disappeared. @zepumph - do want a separated fs for /tmp permanently? Or would you prefer to just keep that on the root LVM (i.e. / in df -h)?

mattpen commented 1 year ago

@zepumph and I added a permanent fs for /tmp in /etc/fstab:

tmpfs           /tmp            tmpfs   rw,mode=1777,size=12g

This mounted without a hitch using mount -a. This seems to be working nicely, and should persist a reboot.