Our RhodeCode instance dies every so often when it runs out of file descriptors. It's leaking the read end of pipes. I found that the issue has been reported and already fixed in Kallithea, a fork of RhodeCode, by Mads Kiilerich, at https://bitbucket.org/conservancy/kallithea/issues/32. Here's a test case that demonstrates the problem, with one extra leaked pipe fd per iteration.
martind@swiftboat:~/download/git_http_backend.py$ curl <https://bitbucket.org/conservancy/kallithea/commits/2f18ebfead0cd4bb2589d2fc734487eb14a499f6/raw> | todos | patch --binary
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 881 100 881 0 0 1241 0 --:--:-- --:--:-- --:--:-- 1242
patching file subprocessio.py
Hunk #1 succeeded at 362 (offset -24 lines).
Hunk #2 succeeded at 389 (offset -25 lines).
martind@swiftboat:~/download/git_http_backend.py$
... then repeat the test, the number of fds is stable, showing that the leak is plugged:
Perhaps I should raise it in its own issue, but my coworker, Uma Parvathappa, discovered that applying the above fix on its own seems to make it easier to expose another frailty. If we create a file larger than the buffer_size:
martind@swiftboat:~/download/git_http_backend.py$ dd if=/dev/zero of=test.txt count=65 bs=1024
65+0 records in
65+0 records out
66560 bytes (67 kB) copied, 0.000401554 s, 166 MB/s
martind@swiftboat:~/download/git_http_backend.py$
... then repeat the previous test of the code with the Kallithea patch, passing it this file as input, then we see an intermittent leak of the write end of pipes along with threads (counted by the second line output for each iteration):
Our RhodeCode instance dies every so often when it runs out of file descriptors. It's leaking the read end of pipes. I found that the issue has been reported and already fixed in Kallithea, a fork of RhodeCode, by Mads Kiilerich, at https://bitbucket.org/conservancy/kallithea/issues/32. Here's a test case that demonstrates the problem, with one extra leaked pipe fd per iteration.
If I apply Mads's patch:
... then repeat the test, the number of fds is stable, showing that the leak is plugged:
Perhaps I should raise it in its own issue, but my coworker, Uma Parvathappa, discovered that applying the above fix on its own seems to make it easier to expose another frailty. If we create a file larger than the buffer_size:
... then repeat the previous test of the code with the Kallithea patch, passing it this file as input, then we see an intermittent leak of the write end of pipes along with threads (counted by the second line output for each iteration):
If, however, we make StreamFeeder do its
os.close(t)in a try... finally block, then we no longer leak, neither pipes nor threads: