Open vbabiy opened 1 year ago
fanzeyi
commented
1 year ago What do you get if you run ulimit -n in the docker?
I think there might be some code that keeps closing the entire fd space (1..max) and your distribution have it set to a much higher value so it just keeps doing that forever.
vbabiy
commented
1 year ago @fanzeyi :point_down:
docker run -it icalialabs/watchman:latest sh
/ # ulimit -n
1073741816
/ #time docker run -it --ulimit nofile=20000 icalialabs/watchman:latest watchman version
{
"version": "4.9.0"
}
docker run -it --ulimit nofile=20000 icalialabs/watchman:latest watchman 0.02s user 0.02s system 7% cpu 0.499 totalThat seems help.
fanzeyi
commented
1 year ago That's good to hear. Also can you try if you can get this reproduced with the latest release? 4.9.0 is a very old version. Please refer to our documents for installations guide: https://facebook.github.io/watchman/docs/install#linux
fanzeyi
commented
1 year ago FWIW we don't own the docker image you are using. That seems to be maintained at https://github.com/IcaliaLabs/docker-watchman
vbabiy
commented
1 year ago @fanzeyi :point_down:
docker run -it --ulimit nofile=20000 gcr.io/fleetchaser/base/python:3.11.2 bash
root@9a0e345c7d19:/# uname -n
9a0e345c7d19
root@9a0e345c7d19:/# time watchman version
version: 20230402.003234.0
buildinfo: 46f701ddb7eab9f8e63b29ddcd5dfd367fdaf699
real 0m0.016s
user 0m0.003s
sys 0m0.003s@fanzeyi this works now, but now I am running in to challenges with #970
vbabiy
commented
1 year ago @fanzeyi this is the change that broke:
When it worked:
sudo systemctl show docker | grep -i NOFILE
LimitNOFILE=1048576
LimitNOFILESoft=1048576Why it doesn't:
sudo systemctl show docker | grep -i NOFILE
LimitNOFILE=infinity
LimitNOFILESoft=infinity
chadaustin
commented
1 year ago @fanzeyi I wonder if it's worth adding a knob to turn off the close-inherited-fds logic.
Or set a maximum fd to try.
fanzeyi
commented
1 year ago @chadaustin yeah we can probably add that. I assume this is where the cost is from? https://github.com/facebook/watchman/blob/126b0cd383df08e806cc1b168e1b86a156ef5a1a/watchman/main.cpp#L205
I feel if we fix the daemon lifecycle management entirely we probably don't need to deal with this.
chadaustin
commented
1 year ago Correct on both counts. Fixing the daemon lifecycle management goes a long way, though I think we still need to support environments where we may not have a reasonable service manager to use, so attacking the problem both ways seems valuable.
On the other hand, Watchman could just leave inherited descriptors open by default and to hell with programs that leak them. :P
In the latest update on Manjaro, I noticed watchman in docker taking forever to start.
Before doing the update this command was instant:
After the update:
htopshows one core pinned:stracehas an endless list of bad file descriptor errors:dockeranddocker composewatchmanDoes anyone have any idea what could cause this?
Here are the full update changes: