Open lucasfcnunes opened 1 year ago
That's disturbing 🤔
I am curious to know if this is actually specific to running micromamba
within the container. I would be interested in knowing if the same output occurs with micromamba
installed directly into Fedora or Arch.
I am curious to know if this is actually specific to running
micromamba
within the container. I would be interested in knowing if the same output occurs withmicromamba
installed directly into Fedora or Arch.
It does work outside of the container. I edited the issue comment.
@lucasfcnunes - on Arch and Fedora, can you please try:
docker run --rm --ulimit nofile=65536:65536 mambaorg/micromamba micromamba run date
thanks!
It did work!
$ docker run --rm --ulimit nofile=65536:65536 mambaorg/micromamba micromamba run date
Fri Aug 25 03:05:45 UTC 2023
# pwsh
(base) PS C:\Users\lucas> ssh lucasfcnunes@archlinux -- docker run --rm --ulimit nofile=65536:65536 mambaorg/micromamba micromamba run date
Fri Aug 25 03:57:41 UTC 2023
(base) PS C:\Users\lucas> ssh lucasfcnunes@fedora -- docker run --rm --ulimit nofile=65536:65536 mambaorg/micromamba micromamba run date
Fri Aug 25 03:57:45 UTC 2023
This solution has become a fairly common one. @jonashaag - do you have any thoughts on changes we could make to have micromamba
produce more helpful error messages when the file handle limit has been reached?
Happy to merge a PR :) I guess you'd have to track down this specific instance of error to find the right place in the code
Workarounds on docker container deployment:
If you don't provide a hard limit value, Docker uses the soft limit value for both values. If you don't provide any values, they are inherited from the default
ulimits
set on the daemon.
docker run --ulimit nofile=65536 ...
# or, equivalently
docker run --ulimit nofile=65536:65536 ...
# compose.yaml
services:
myservice:
...
ulimits:
nofile:
soft: 65536
# hard: <default = soft>
...
@wholtz @jonashaag Question: Why specifically nofile and soft=65536 and soft=hard?
I was expecting to reproduce the Success
message when tweaking the numbers, but docker itself gave me another error (on nofile=12 -> 11).
(base) PS C:\Users\lucas> ssh lucasfcnunes@archlinux -- docker run --ulimit nofile=12 --rm mambaorg/micromamba micromamba run date
Sat Aug 26 19:57:32 UTC 2023
(base) PS C:\Users\lucas> ssh lucasfcnunes@archlinux -- docker run --ulimit nofile=11 --rm mambaorg/micromamba micromamba run date
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error loading seccomp filter into kernel: error patching filter: error disassembling original filter: error creating scratch pipe: pipe2: too many open files: unknown.
NativeCommandExitException: Program "ssh.exe" ended with non-zero exit code: 125.
65536 is expected to be "big enough" for many applications. I don't recall the details, but there can be a performance hit if you set nofile too high, so you don't want to just max out the nofile value.
We could have set different hard and soft values. I like to keep them the same as it reduces the complexity a little bit.
If you make nofile small enough (such as 11), you are changing the point in the code path where the system runs out of file handles.
After some experiments seems that:
nofile
> 2^20+1 causes the print of the Success
messagenofile
> 2^20nofile
s are (1048576, 1073741816, 1024) = (ubuntu, arch, fedora)nofile
s are (1024, 1024, 1024) = (ubuntu, arch, fedora)(base) PS C:\Users\lucas\projects\micromamba_os> task os:test:all
task: [os:test] ssh lucasfcnunes@ubuntu -- set -x '&&' hostnamectl '&&' ulimit -n '&&' docker run --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true '&&' docker run --ulimit nofile=1048576 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true '&&' docker run --ulimit nofile=1048577 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true '&&' docker run --ulimit nofile=1048578 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&"
micromamba run date\' '||' true
+ hostnamectl
Static hostname: ubuntu
Icon name: computer-vm
Chassis: vm
Machine ID: db7f2b6affe44ea6b3e550c6b865c49f
Boot ID: 6b71703eb2b34f3681d511c3f2d8e3f3
Virtualization: microsoft
Operating System: Ubuntu 22.04.3 LTS
Kernel: Linux 6.2.0-26-generic
Architecture: x86-64
1024
+ ulimit -n
+ docker run --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
1048576
Sun Aug 27 12:57:13 UTC 2023
+ docker run --ulimit nofile=1048576 --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
1048576
Sun Aug 27 12:57:14 UTC 2023
+ docker run --ulimit nofile=1048577 --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting rlimits for ready process: error
setting rlimit type 7: operation not permitted: unknown.
+ true
+ docker run --ulimit nofile=1048578 --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting rlimits for ready process: error
setting rlimit type 7: operation not permitted: unknown.
+ true
task: [os:test] ssh lucasfcnunes@arch -- set -x '&&' hostnamectl '&&' ulimit -n '&&' docker run --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true '&&' docker run --ulimit nofile=1048576 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true '&&' docker run --ulimit nofile=1048577 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true '&&' docker run --ulimit nofile=1048578 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true
+ hostnamectl
Static hostname: arch
Icon name: computer-vm
Chassis: vm 🖴
Machine ID: 26408a174bed44cf82501d8c859e1e46
Boot ID: c72088e41961464dbf14c003e8f5f9a9
Virtualization: microsoft
Operating System: Arch Linux
Kernel: Linux 6.1.46-1-lts
Architecture: x86-64
Hardware Vendor: Microsoft Corporation
Hardware Model: Virtual Machine
Firmware Version: Hyper-V UEFI Release v4.0
Firmware Date: Fri 2019-11-01
Firmware Age: 3y 9month 3w 4d
1024
+ ulimit -n
+ docker run --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
1073741816
Success
+ true
+ docker run --ulimit nofile=1048576 --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
1048576
Sun Aug 27 12:57:18 UTC 2023
+ docker run --ulimit nofile=1048577 --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
1048577
Sun Aug 27 12:57:20 UTC 2023
+ docker run --ulimit nofile=1048578 --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
1048578
Success
+ true
task: [os:test] ssh lucasfcnunes@fedora -- set -x '&&' hostnamectl '&&' ulimit -n '&&' docker run --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true '&&' docker run --ulimit nofile=1048576 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true '&&' docker run --ulimit nofile=1048577 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true '&&' docker run --ulimit nofile=1048578 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&"
micromamba run date\' '||' true
+ hostnamectl
Static hostname: fedora
Icon name: computer-vm
Chassis: vm 🖴
Machine ID: 6690c3cc7b80464aac61602ff30933a1
Boot ID: c6baa5d0563c40f2a285ba55fbbeec33
Virtualization: microsoft
Operating System: Fedora Linux 38 (Workstation Edition)
CPE OS Name: cpe:/o:fedoraproject:fedora:38
OS Support End: Tue 2024-05-14
OS Support Remaining: 8month 2w 2d
Kernel: Linux 6.4.12-200.fc38.x86_64
Architecture: x86-64
Hardware Vendor: Microsoft Corporation
Hardware Model: Virtual Machine
Firmware Version: Hyper-V UEFI Release v4.0
Firmware Date: Fri 2019-11-01
1024
+ ulimit -n
+ docker run --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
1024
Sun Aug 27 12:57:23 UTC 2023
+ docker run --ulimit nofile=1048576 --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
1048576
Sun Aug 27 12:57:25 UTC 2023
+ docker run --ulimit nofile=1048577 --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
1048577
Sun Aug 27 12:57:28 UTC 2023
+ docker run --ulimit nofile=1048578 --rm mambaorg/micromamba -- sh -c 'ulimit -n && micromamba run date'
1048578
Success
+ true
# Taskfile.yaml
version: "3"
output: group
tasks:
os:test:
cmds:
- >-
ssh {{.HOSTNAME}} -- set -x '&&' hostnamectl
'&&' docker run --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true
'&&' docker run --ulimit nofile=1048576 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true
'&&' docker run --ulimit nofile=1048577 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true
'&&' docker run --ulimit nofile=1048578 --rm mambaorg/micromamba -- sh -c \'ulimit -n "&&" micromamba run date\' '||' true
os:test:all:
cmds:
- task: os:test
vars:
HOSTNAME: lucasfcnunes@ubuntu
- task: os:test
vars:
HOSTNAME: lucasfcnunes@arch
- task: os:test
vars:
HOSTNAME: lucasfcnunes@fedora
Can anyone try the static build of this PR: #2848 (once it's built)
I'd like to know if there is a specific error code reported that we could inspect
bump
Can anyone try the static build of this PR: #2848 (once it's built)
I'd like to know if there is a specific error code reported that we could inspect
@jonashaag did it work? I tried to find https://github.com/mamba-org/mamba/pull/2848/commits/73f1eae9185f196da6493b34c6cd019c9e07c9fe on docker hub but nothing returned https://hub.docker.com/r/mambaorg/micromamba/tags?page=1&name=73f1e
See here on the bottom of the page https://github.com/mamba-org/mamba/actions/runs/6224813476?pr=2848
Troubleshooting docs
How did you install Mamba?
Mambaforge
Search tried in issue tracker
Done
Latest version of Mamba
Tried in Conda?
I didn't try
Describe your issue
micromamba run
doesn't work as expected. It prints 'Success' on any command. It's the case in:It works as expected on Ubuntu hosts.
mamba info / micromamba info
Logs
No response
environment.yml
No response
~/.condarc
No response