Open yyc opened 4 weeks ago
Hey @yyc, I haven't reproduced yet, and I'm not sure what, if anything, we would be running that is using /tmp concurrently with stack setup. However, I have an idea for a workaround in #1327. WDYT?
Hey @yyc, I haven't reproduced yet, and I'm not sure what, if anything, we would be running that is using /tmp concurrently with stack setup. However, I have an idea for a workaround in #1327. WDYT?
@DrJosh9000 Yes I've tested that and it looks like it works! Thanks for the quick fix :)
Describe the bug
MountTmpfsAtTemp=false
doesn't always seem to take effect. A large proportion of instances come up withtmpfs
still mounted to/tmp
, and then get terminated.This causes a lot of instance churn on scale-up.
I suspect there is something else running on startup that writes to the
/tmp
dir at around the same time thatbk-mount-instance-storage.sh
is run, blocking the unmount operation and causing it to fail. Is there anything in the AMI that might do that?note: it's entirely possible that there's something in our custom AMI that we build atop of the buildkite-provided one that is responsible for this. If you can't reproduce this, that is still good info that we need to audit our startup processes
Steps To Reproduce Steps to reproduce the behavior:
v6.21.0
MountTmpfsAtTemp
to falseInService
state, but get terminated after ~1 minute.Expected behavior when
MountTmpfsAtTemp
is set to false, the instance runssystemctl mask --now tmp.mount
on startup, which correctly unmounts tmpfs from the/tmp
directory. We observe this in a small number of instances that come up:Actual behaviour On a large number of instances, instead we see the following:
Stack parameters:
ami-04ca34320055d861c