System.IO.IOException: No space left on device

zspitzer commented 2 years ago

Description

Run df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/root         11M 1002K  9.7M   10% /
devtmpfs         8[6](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:7)[7](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:8)K   450  [8](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:9)67K    1% /dev
tmpfs            868K     5  868K    1% /dev/shm
tmpfs            868K   75[9](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:10)  868K    1% /run
tmpfs            868K     3  868K    1% /run/lock
tmpfs            868K    19  868K    1% /sys/fs/cgroup
/dev/loop0        486   486     0  [10](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:11)0% /snap/snapd/16292
/dev/loop2        [12](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:13)K   12K     0  100% /snap/core20/1518
/dev/loop1        802   802     0  100% /snap/lxd/22753
/dev/sdb15          0     0     0     - /boot/efi
/dev/sda1        896K    [13](https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true#step:2:14)  896K    1% /mnt
tmpfs            868K    22  868K    1% /run/user/1001

https://github.com/lucee/extension-compress/runs/7465171399?check_suite_focus=true

Platforms affected

[ ] Azure DevOps
[X] GitHub Actions

Virtual environments affected

[ ] Ubuntu 18.04
[X] Ubuntu 20.04
[ ] Ubuntu 22.04
[ ] macOS 10.15
[ ] macOS 11
[ ] macOS 12
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link

Virtual Environment Environment: ubuntu-20.04 Version: 20220717.1 Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20220717.1/images/linux/Ubuntu2004-Readme.md Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F20220717.1

Is it regression?

yes

Expected behavior

runner should have more than 9.7MB free on /

Actual behavior

build Unhandled exception. System.IO.IOException: No space left on device : '/home/runner/runners/2.294.0/_diag/Worker_20220722-085135-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan1 buffer, Int64 fileOffset) at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite() at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.Diagnostics.TextWriterTraceListener.Flush() at System.Diagnostics.TraceSource.Flush() at GitHub.Runner.Common.TraceManager.Dispose(Boolean disposing) at GitHub.Runner.Common.TraceManager.Dispose() at GitHub.Runner.Common.HostContext.Dispose(Boolean disposing) at GitHub.Runner.Common.HostContext.Dispose() at GitHub.Runner.Worker.Program.Main(String[] args) System.IO.IOException: No space left on device : '/home/runner/runners/2.294.0/_diag/Worker_20220722-085135-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan1 buffer, Int64 fileOffset) at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite() at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.Diagnostics.TextWriterTraceListener.Flush() at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id) at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message) at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message) at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut) at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args) System.IO.IOException: No space left on device : '/home/runner/runners/2.294.0/_diag/Worker_20220722-085135-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset) at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite() at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder) at System.Diagnostics.TextWriterTraceListener.Flush() at GitHub.Runner.Common.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id) at GitHub.Runner.Common.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message) at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message) at GitHub.Runner.Common.Tracing.Error(Exception exception) at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)

Repro steps

run the work flow https://github.com/lucee/extension-compress/actions/workflows/main.yml

mikhailkoliada commented 2 years ago

@zspitzer hi!

We will take a look

mikhailkoliada commented 2 years ago

@zspitzer hello!

I see that:

we do not have fs quota enabled on runners
you still have enough free space before tests run (I mean plain df -h)
even if I lift ext4 fs reservation to 0% via tune2fs -m 0 the problem is still existing So my only guess is that space is exhausted somewhere in the middle of the failed step. Could you try to attach strace to the run itself or the process so it is possible to save logs and see what happens under the hood?

mikhailkoliada commented 2 years ago

I am going to close this issue as we do not have any updates for the time being, feel free to contact us again if you have more details or any questions left.

actions / runner-images