ionelmc
commented
3 years ago Are there some memory limits or something that may make the redis server close connections?
Open mcepl opened 3 years ago
ionelmc
commented
3 years ago Are there some memory limits or something that may make the redis server close connections?
mcepl
commented
3 years ago Yes, probably, but they should be pretty high: my computer has 15 GB RAM, 700+ MB free, and there are some constraints in the build system, but those should be again in GBs of RAM.
ionelmc
commented
3 years ago Maybe you have some file descriptor limits that create that disconnect problem?
mcepl
commented
3 years ago Maybe you have some file descriptor limits that create that disconnect problem?
$ ulimit -a
real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63393
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 63393
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$open files = 1024 is not really adequate, there'd be lots of fd trashing in the ton of integration tests that PRL has.
mcepl
commented
3 years ago open files = 1024 is not really adequate, there'd be lots of fd trashing in the ton of integration tests that PRL has.
Rising the value to 63393 doesn't seem to help too much.
ionelmc
commented
3 years ago Well there's something weird going on. I'd need to change a bit the tests to dump out the redis logs. But you're the only one able to reproduce this problem so I must ask if this is really worth pursuing - running integration tests in a build system...
mcepl
commented
3 years ago On the one hand of course adding one exception to pytest call is not a big deal. On the other hand, I like to run as many tests as possible to (hopefully) avoid too many regressions. I can understand if it is just not possible to fix this, I just wonder.
ionelmc
commented
3 years ago Well it's suspicious that it fails just that one test (I hope you have tried to run the tests at least a dozen times). I'm not saying it's not possible to fix (or better said, to figure out why it happens), but it'll take time. CI passed 20 days ago (https://travis-ci.org/ionelmc/python-redis-lock) so I asked if it's worth spending the time.
mcepl
commented
3 years ago It is probably not. Just leave this ticket open, and it can be fixed anytime.
ionelmc
commented
3 years ago It's up to you ;-)
Try this patch:
diff --git a/tests/test_redis_lock.py b/tests/test_redis_lock.py
index 5d91e8e..0915543 100644
--- a/tests/test_redis_lock.py
+++ b/tests/test_redis_lock.py
@@ -364,7 +364,7 @@ NWORKERS = 125
@pytest.mark.skipif(platform.python_implementation() == 'PyPy', reason="This appears to be way too slow to run on PyPy")
-def test_no_overlap2(make_process, make_conn):
+def test_no_overlap2(make_process, make_conn, redis_server):
"""The second version of contention test, that uses multiprocessing."""
go = multiprocessing.Event()
count_lock = multiprocessing.Lock()
@@ -397,6 +397,7 @@ def test_no_overlap2(make_process, make_conn):
time.sleep(1)
+ print(redis_server.read())
assert count.value == 1
The relevant part is probably:
[ 197s] redis_server = 6403:C 25 Mar 2021 20:17:24.669 # oO0OoO0OoO0Oo Redis is starting o
O0OoO0OoO0Oo
[ 197s] 6403:C 25 Mar 2021 20:17:24.669 # Redis version=6.2.1, bits=64, commit=00000000, m
odified=0, pid=6403, just started
[ 197s] 6403:C 25 Mar 2021 20:17:24.669 # Configuration loaded
[ 197s] 6403:M 25 Mar 2021 20:17:24.669 * monotonic clock: POSIX clock_gettime
[ 197s] 6403:M 25 Mar 2021 20:17:24.669 * Running mode=standalone, port=0.
[ 197s] 6403:M 25 Mar 2021 20:17:24.669 # Server initialized
[ 197s] 6403:M 25 Mar 2021 20:17:24.670 * Loading RDB produced by version 6.2.1
[ 197s] 6403:M 25 Mar 2021 20:17:24.670 * RDB age 0 seconds
[ 197s] 6403:M 25 Mar 2021 20:17:24.670 * RDB memory usage when created 3.27 Mb
[ 197s] 6403:M 25 Mar 2021 20:17:24.670 * DB loaded from disk: 0.000 seconds
[ 197s] 6403:M 25 Mar 2021 20:17:24.670 * The server is now ready to accept connections at
/tmp/redis-lock-tests.sockBy the way, talking about complicated and non-complicated fixes:
[ 197s] /home/abuild/rpmbuild/BUILD/python-redis-lock-3.6.0/tests/conftest.py:12: Pytest
DeprecationWarning: @pytest.yield_fixture is deprecated.
[ 197s] Use @pytest.fixture instead; they are the same.
[ 197s] def redis_server(scope='session'):
Bubu
commented
2 years ago I'm also getting this now in Archlinux. (And Nix apperently as well as can be seen above).
This didn't happen when I built the package on Fri 11 Dec 2020, so there might be some change in the ecosystem between Dec 2020 and Mar '21. :thinking:
Bubu
commented
1 year ago I just upgraded the archlinux package to 4.0.0. The test in question here passed with no problems.
When packaging this package for openSUSE/Factory I get this error:
Complete build log with all details and versions of all packages used.