Open konradpabjan opened 6 months ago
Also seen in my workflow.
Run actions/upload-artifact@v4.0.0
With the provided path, there will be [7](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:8)3166 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
node:events:492
throw er; // Unhandled 'error' event
^
Error: EMFILE: too many open files, open '/home/runner/work/prima/prima/prima_prima_quadruple.1_20.ubln.single.231227_1211_start/HS63'
Emitted 'error' event on ReadStream instance at:
at emitErrorNT (node:internal/streams/destroy:151:[8](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:9))
at emitErrorCloseNT (node:internal/streams/destroy:[11](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:12)6:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -24,
code: 'EMFILE',
syscall: 'open',
path: '/home/runner/work/prima/prima/prima_prima_quadruple.1_20.ubln.single.23[12](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:13)27_1211_start/HS63'
}
Node.js v[20](https://github.com/primalib/prima/actions/runs/7338125515/job/19983281711#step:4:21).8.1
We can possibly fix with the file
method in archiver:
Appends a file given its filepath using a lazystream wrapper to prevent issues with open file limits.
Right now we are just using the append
operation and creating a read stream per file:
Bumping as I'm facing the same issue on a repo :)
Same issue here when I tried to upload 9,834 files for a single artifact. The compression-level
made no difference.
Had to downgrade to upload-artifact@v3
and download-artifact@v3
.
actions/upload-artifact@main
with:
name: MSYS
path: C:\MSYS
if-no-files-found: warn
compression-level: 6
overwrite: false
env:
MSYSTEM: MINGW64
With the provided path, there will be 51574 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
node:events:492
throw er; // Unhandled 'error' event
^
Error: EMFILE: too many open files, open 'C:\MSYS\msys64\mingw64\lib\python3.11\distutils\tests\__pycache__\test_cygwinccompiler.cpython-311.pyc'
Emitted 'error' event on ReadStream instance at:
at emitErrorNT (node:internal/streams/destroy:151:8)
at emitErrorCloseNT (node:internal/streams/destroy:116:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -4066,
code: 'EMFILE',
syscall: 'open',
path: 'C:\\MSYS\\msys64\\mingw64\\lib\\python3.11\\distutils\\tests\\__pycache__\\test_cygwinccompiler.cpython-311.pyc'
}
Node.js v20.8.1
same bug at latest https://github.com/actions/upload-artifact/tree/3a8048248f2f288c271830f8ecf2a1c5d8eb0e9a
actions/upload-artifact@v3 is good but warning
Warning: There are over 10,000 files in this artifact, consider creating an archive before upload to improve the upload performance.
Is this fixed yet?
Running into this with Linux too: https://github.com/ehuelsmann/LedgerSMB/actions/runs/8121634001/job/22200227067#step:6:26
Ditto, just ran into the same problem :/
I've switched https://github.com/actions/toolkit/pull/1723 (proposed fix for retaining file permissions in uploaded .zip files) to use zip.file(file.sourcePath)
instead of zip.append(createReadStream(file.sourcePath))
so that that PR can also fix this bug at the same time. It needs a review from repo maintainers before it can go anywhere, though.
Is there a command that we can run in the docker container that allow us to increment the number of open file descriptors to 10k lets say for example??
Wouldn't that fix this issue?
Just found got this issue. Using v3 I get warnings its going to be killed soon aswell:
https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/
This needs to be fixed before that happens @konradpabjan @robherley otherwise it will make it not possible to upload artifacts that contain over the OS file descriptor limit.
My workaround is to compress first to a .tgz
, then upload it. A clunky approach, but seems to do the trick for me. However, I suppose this might be a non-solution for some.
oneJob:
steps:
- name: 🏗 Checkout branch
uses: actions/checkout@v4
# ...
- name: 🗜️ Compress
run: tar -czf ./upload.tar.gz ./build
- name: ⬆️ Upload results
uses: actions/upload-artifact@v4
with:
name: build-${{ github.event.number }}
path: ./upload.tar.gz
overwrite: true
if-no-files-found: error
retention-days: 1 # No need to keep these longer for now, until we can figure if there's a way to share these between jobs
compression-level: 0 # We are already gzipping above, makes no sense to compress again. Remove wehen upload-artifact fixes its stuff
twoJob:
needs: oneJob
steps:
- name: 🏗 Checkout branch
uses: actions/checkout@v4
- name: ⬇️ Download results
uses: actions/download-artifact@v4
with:
name: build-${{ github.event.number }}
- name: 🤯 Expand
run: tar -xzf ./upload.tar.gz
@konradpabjan - https://github.com/actions/toolkit/pull/1723 is a PR that should fix this bug (along with https://github.com/actions/upload-artifact/issues/38); it only needs a review from a GitHub team member before it can be merged. Would you have time to take a look at it? It's been nearly two months since I opened the PR, and I'd really like it to not sit there unreviewed for another two months.
Please merge this before v3 support is removed from GitHub cloud :(
What happened?
Stumbled on a failure that happened with v4. Apparently this doesn't happen with v3
Example run can be found here: https://github.com/spacetelescope/jdaviz/actions/runs/7266578163/job/19798598137
What did you expect to happen?
Successful artifact upload
How can we reproduce it?
Not sure, but I think the key here is a large number of files. In the logs it shows
8824
files which is quite a bit. Maybe some read stream isn't being closed somewhere 🤔Anything else we need to know?
Nope
What version of the action are you using?
v4
What are your runner environments?
windows
Are you on GitHub Enterprise Server? If so, what version?
No