t/memcached test fails occasionally in Travis CI

[esabol]

The t/memcached fails in Travis CI sometimes. I figured we needed an issue to track it.

When the failure happens, the following message appears in test-suite.log:

libtest/client.cc:269: in start() pid(30680) localhost:60040 ping(libtest/client.cc:269: Connection refused), additionally pid: 30745 is alive: true waited: 17 server started. exec: /home/travis/build/gearman/gearmand/libtool --mode=execute /home/travis/build/gearman/gearmand/./gearmand/gearmand --verbose=INFO --log-file=var/log/gearmand.loggBEQRs --pid-file=var/run/gearmand.pidkQkA2d --port=60040 --queue-type=libmemcached --libmemcached-servers=localhost:24735 --listen=localhost  stderr:
tests/memcached_test.cc:119: in lp_1054377_TEST() pid(30680) Assertion 'server_startup(servers, "gearmand", first_port, argv)'
memcached.queue restart.lp:1054377                  [ failed ]

What causes it?

A race condition with libtest's get_free_port(), right? So the port becomes busy in between the call to get_free_port() and when the t/memcached uses it, presumably.

Possible solutions

• Disable memcached in Travis CI? Effective, but not desirable since it reduces code coverage.
• Modify the test to retry a few times?
• Randomize the port number?
• Other ideas?

[SpamapS]

Related/duplicate https://github.com/gearman/gearmand/issues/50

[esabol]

Ah, right... I saw that, but I didn't get past the title since I was mainly concerned with t/memcached instead t/cycle. Lots of good discussion there already.

[esabol]

Ok, how about having memcached bind to port 0 and then using ss or lsof or netstat to get the port number memcached ends up listening on?

https://unix.stackexchange.com/questions/157823/list-ports-a-process-pid-is-listening-on-preferably-using-iproute2-tools

[esabol]

Actually, it appears the problem isn't with the memcached port, but in starting up gearmand: https://github.com/gearman/gearmand/blob/c343b3b93e057dccf2a93a8109663437ceded148/tests/memcached_test.cc#L151

[p-alik]

Actually, it appears the problem isn't with the memcached port, but in starting up gearmand

Indeed, but in the line 119 https://github.com/gearman/gearmand/blob/c343b3b93e057dccf2a93a8109663437ceded148/tests/memcached_test.cc#L119