FAIL: test_get_collection_ttl on make test under Debian 8

[qirtaiba]

I can't successfully "make test" on a standard Debian Jessie system from the latest (yesterday) git sources. The first error I got is the one at issue #61 and I patched that one. But now I get this:

$ make test
./local/bin/flake8 ./syncserver
./local/bin/nosetests -s syncstorage.tests
...................................................................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS............................................SSSSSSSSSSSSSSSS.SS................
----------------------------------------------------------------------
Ran 281 tests in 339.087s

OK (SKIP=153)
# Tokenserver tests currently broken due to incorrect file paths
# ./local/bin/nosetests -s tokenserver.tests
# Test against a running server
./local/bin/gunicorn --paste syncserver/tests.ini 2> /dev/null & SERVER_PID=$!; \
sleep 2; \
./local/bin/python -m syncstorage.tests.functional.test_storage \
    --use-token-server http://localhost:5000/token/1.0/sync/1.5; \
kill $SERVER_PID
s...s.......F........s...................s
======================================================================
FAIL: test_get_collection_ttl (syncstorage.tests.functional.support.LiveTestCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/syncserver/local/lib/python2.7/site-packages/syncstorage/tests/functional/test_storage.py", line 714, in test_get_collection_ttl
    self.assertEquals(len(res.json), 1)
AssertionError: 0 != 1

----------------------------------------------------------------------
Ran 42 tests in 225.623s

FAILED (failures=1, skipped=4)

[rfk]

Does this happen every time, or only sporadically? It looks like a timestamp-related test which, if it's using the system clock, could fail just due to vagaries of system behaviour on slower machines.

[qirtaiba]

Every time on a lightly loaded (but quite slow, yes) virtual machine with 8Gb of RAM. But it doesn't seem to be the same error each time. This time was:

$ make test
./local/bin/flake8 ./syncserver
./local/bin/nosetests -s syncstorage.tests
...................................................................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS............................................SSSSSSSSSSSSSSSS.SS................
----------------------------------------------------------------------
Ran 281 tests in 332.909s

OK (SKIP=153)
# Tokenserver tests currently broken due to incorrect file paths
# ./local/bin/nosetests -s tokenserver.tests
# Test against a running server
./local/bin/gunicorn --paste syncserver/tests.ini 2> /dev/null & SERVER_PID=$!; \
sleep 2; \
./local/bin/python -m syncstorage.tests.functional.test_storage \
    --use-token-server http://localhost:5000/token/1.0/sync/1.5; \
kill $SERVER_PID
s...sF...............s...................s
======================================================================
FAIL: test_bulk_update_of_ttls_without_sending_data (syncstorage.tests.functional.support.LiveTestCases)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/syncserver/local/lib/python2.7/site-packages/syncstorage/tests/functional/test_storage.py", line 1111, in test_bulk_update_of_ttls_without_sending_data
    self.assertEquals(sorted(items.keys()), ["3", "4", "5", "6"])
AssertionError: Lists differ: [] != ['3', '4', '5', '6']

Second list contains 4 additional elements.
First extra element 0:
3

- []
+ ['3', '4', '5', '6']

----------------------------------------------------------------------
Ran 42 tests in 209.115s

FAILED (failures=1, skipped=4)

[rfk]

Looks like it's almost certainly a timing issue with the tests. This codebase is pretty stable and is only receiving maintenance patches at the moment, with the folks behind it focusing on other things, so I'm not sure how likely we are to be able to prioritize a fix for this. While I feel bad doing it, I suggest just ignoring the failing tests and proceeding to run the server since it looks like things are actually working fine.

[rfk]

Closing out this old issue, I don't think we're going to make any further followups here.