Unexpected behavior with "noreply"

[mckelvin]

The issue can be reproduced as follows:

$ cat ~/reproduce.py
import socket
import time

def main():
    val = str(int(time.time()))
    sock = socket.socket()
    sock.connect(('localhost', 8102))
    sock.send('delete foo noreply\r\n')
    sock.send('set foo 0 0 %d noreply\r\n%s\r\n' % (len(val), val))
    sock.send('get foo\r\n')
    resp = sock.recv(1024)
    try:
        assert 'foo' in resp
    except:
        print 'unexpected response: %s' % resp.replace('\r', r'\r').replace('\n', r'\n')
        raise
    finally:
        sock.close()

if __name__ == '__main__':
    main()
$ python ~/reproduce.py
unexpected response: END\r\n
Traceback (most recent call last):
  File "/home/mckelvin/reproduce.py", line 23, in <module>
    main()
  File "/home/mckelvin/reproduce.py", line 14, in main
    assert 'foo' in resp
AssertionError
$ python ~/reproduce.py
$ python ~/reproduce.py
unexpected response: END\r\n
Traceback (most recent call last):
  File "/home/mckelvin/reproduce.py", line 23, in <module>
    main()
  File "/home/mckelvin/reproduce.py", line 14, in main
    assert 'foo' in resp
AssertionError

I also did the same test on facebook/mcrouter and twitter/twemproxy. This will only happen on Netflix/dynomite. So I guess that's a bug.

[mckelvin]

@timiblossom can you confirm the issue?

[timiblossom]

Hi Kelvin.

Thank you for reporting this. I will investigate this. If it is, it is a serious bug that I need to fix immediately.

Did you run this against a single node Dynomite cluster? Otherwise, what was your setup?

Thanks. Minh

[mckelvin]

Did you run this against a single node Dynomite cluster?

Yes. And here is the setup:

The dynomite is built inside a Docker image

FROM phusion/baseimage:0.9.15

MAINTAINER mckelvin <xxx@gmail.com>
ENV DEBIAN_FRONTEND noninteractive
RUN sed -i 's/archive.ubuntu.com/mirrors.163.com/g' /etc/apt/sources.list
RUN apt-get update && apt-get install -yq \
    libevent-dev \
    autoconf \
    libtool \
    make \
    libssl-dev
ADD dynomite /tmp/dynomite
RUN cd /tmp/dynomite && \
    autoreconf -fvi && CFLAGS="-O3" ./configure --enable-debug=log && \
    make && make install && \
    rm -rf /tmp/dynomite

/etc/single.yml:

cat etc/single.yml
dyn_o_mite:
  dyn_listen: 0.0.0.0:8101
  listen: 0.0.0.0:8102
  servers:
  - theoden:21211:1
  tokens: 437425602
  secure_server_option: none
  pem_key_file: /etc/dynomite/dynomite.pem

Finally, the dynomite process runs via: docker run -p 8102:8102 -p 8101:8101 -p 22222:22222 --rm -vpwd/etc:/etc/dynomite docker-registry.intra.douban.com/dynomite dynomite -c /etc/dynomite/single.yml -v 6

[timiblossom]

@mckelvin I ran our python code against Redis itself and get a similar error:

mdo@ubuntu:~/workspace/dynomite/scripts$ python test.py unexpected response: -ERR unknown command 'delete'\r\n-ERR syntax error\r\n-ERR unknown command '1420276404'\r\n$-1\r\n Traceback (most recent call last): File "test.py", line 23, in main() File "test.py", line 14, in main assert 'foo' in resp AssertionError mdo@ubuntu:~/workspace/dynomite/scripts$

Also, I can't find on Redis' set with noreply. Can you point it out to me so I can explore it a bit?

[mckelvin]

@timiblossom Oh, I forget to mention this:the issue can only be reproduced against memcached(instead of Redis). Redis doesn't even support noreply.

[timiblossom]

@mckelvin I confirm that this is a bug on our side and we are working to address it. Thanks

[timiblossom]

Fixed in master branch.

[mckelvin]

:beers: