Open izorster opened 2 years ago
iwilltry42
commented
2 years ago Hi @izorster , thanks for opening this issue! Does this only happen with Calico or also with any other deployment? EDIT: I only ever saw a similar log here: https://k3d.io/v5.3.0/faq/faq/#spurious-pid-entries-in-proc-after-deleting-k3d-cluster-with-shared-mounts
izorster
commented
2 years ago Only happens with calico cube controller of the calico deployment and only on MacOS:
On Mar 22, 2022, at 12:26 PM, Thorsten Klein @.***> wrote:
Hi @izorster https://github.com/izorster , thanks for opening this issue! Does this only happen with Calico or also with any other deployment?
— Reply to this email directly, view it on GitHub https://github.com/k3d-io/k3d/issues/1011#issuecomment-1075360407, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNYZM6NDLR2PKMGEVONA4LVBHYEDANCNFSM5QN535ZA. You are receiving this because you were mentioned.
iwilltry42
commented
2 years ago Maybe it's related to inotify limits in the Docker VM of DfD. See e.g. https://github.com/kubernetes-sigs/kind/issues/2475 which points to https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files Or in general for DfD: https://jhooq.com/docker-error-no-space-left/
izorster
commented
2 years ago I read through and tried some of this…the first link didn’t seem to be the same issue. The link it points to states to increase params on linux-like os (I assume the pods, etc.) which after docker exec-ing in, set, killed off pod to restart resulted in the same issue. The third one is irrelevant as I am using Docker desktop and have allocated 12G of ram and 80G of storage space. I found nowhere to change I-notify for MacOS…
Thanks
On Mar 23, 2022, at 12:08 PM, Thorsten Klein @.***> wrote:
Maybe it's related to inotify limits in the Docker VM of DfD. See e.g. kubernetes-sigs/kind#2475 https://github.com/kubernetes-sigs/kind/issues/2475 which points to https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files https://kind.sigs.k8s.io/docs/user/known-issues/#pod-errors-due-to-too-many-open-files Or in general for DfD: https://jhooq.com/docker-error-no-space-left/ https://jhooq.com/docker-error-no-space-left/ — Reply to this email directly, view it on GitHub https://github.com/k3d-io/k3d/issues/1011#issuecomment-1076518678, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNYZM6A4CSJCGOZNT5T2DLVBM6YBANCNFSM5QN535ZA. You are receiving this because you were mentioned.
iwilltry42
commented
2 years ago True, but you're on Mac.. Docker for Desktop uses a Linux VM under the hood and you can tweak that VM's options as well using a privileged pod: https://stackoverflow.com/questions/39739560/how-to-access-the-vm-created-by-dockers-hyperkit/60582083#60582083
Again: This is trial and error. In any case, it was confirmed multiple times, that Calico works just fine in k3d on various systems :thinking:
izorster
commented
2 years ago The i-notify values are larger than what is suggested in the post leading me to believe that this still is not the issue:
docker-desktop:/etc/sysctl.d# grep notify * 04-inotify.conf:fs.inotify.max_user_watches = 1048576 04-inotify.conf:fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 524288 fs.inotify.max_user_instances = 512
On Mar 24, 2022, at 12:05 PM, Thorsten Klein @.***> wrote:
True, but you're on Mac.. Docker for Desktop uses a Linux VM under the hood and you can tweak that VM's options as well using a privileged pod: https://stackoverflow.com/questions/39739560/how-to-access-the-vm-created-by-dockers-hyperkit/60582083#60582083 https://stackoverflow.com/questions/39739560/how-to-access-the-vm-created-by-dockers-hyperkit/60582083#60582083 Again: This is trial and error. In any case, it was confirmed multiple times, that Calico works just fine in k3d on various systems 🤔
— Reply to this email directly, view it on GitHub https://github.com/k3d-io/k3d/issues/1011#issuecomment-1077790815, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNYZM5CNH24F5K7RXEJT6DVBSHFFANCNFSM5QN535ZA. You are receiving this because you were mentioned.
iwilltry42
commented
2 years ago Unfortunately now I'm out of ideas here. On a different hand: are you using some wrapper around k3d?
--k3d_options "--volume" "$(pwd)/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml"
izorster
commented
2 years ago Yes, we have a script that invokes k3d and options are based on the parameters passed/parsed to the script.
On Mar 25, 2022, at 2:03 PM, Thorsten Klein @.***> wrote:
Unfortunately now I'm out of ideas here. On a different hand: are you using some wrapper around k3d?
--k3d_options "--volume" "$(pwd)/calico.yaml:/var/lib/rancher/k3s/server/manifests/calico.yaml"
— Reply to this email directly, view it on GitHub https://github.com/k3d-io/k3d/issues/1011#issuecomment-1079273171, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUNYZM2M5LQL24XPNHWHH4TVBX5XBANCNFSM5QN535ZA. You are receiving this because you were mentioned.
What did you do
How was the cluster created?
What did you do afterwards?
What did you expect to happen
The cluster and application pods to come up/stay up.
Screenshots or terminal output
If applicable, add screenshots or terminal output (code block) to help explain your problem.
In the calico-kube-controllers pod events:
Which OS & Architecture
MacOS 12
Which version of
k3d5.3.0
Which version of docker