jplana / python-etcd

A python client for etcd
Other
520 stars 210 forks source link

Fix eternal watch cause error 'the requested history has been cleared' #220

Closed mingqing closed 7 years ago

mingqing commented 7 years ago

If client occur reconnection and etcd waitIndex less than event history startIndex will be cause error, like “the requested history has been cleared [6445109/1]”

coveralls commented 7 years ago

Coverage Status

Coverage increased (+0.02%) to 88.462% when pulling 9cc3af3214c2939c1e487f346f116e0c1e0642ff on mingqing:fix-eternal-watch into 0d0145f5e835aa032c97a0a5e09c4c68b7a03f66 on jplana:master.

mingqing commented 7 years ago

@ashcrow

1) create keys and watch it

import etcd

client = etcd.Client(
    allow_reconnect=True,
    host=(
        ("10.5.117.140", 2379),
        ("10.5.117.141", 2379),
        ("10.5.117.142", 2379)
    )
)

client.write("/hello/test", "value")

for event in client.eternal_watch("/hello/test", recursive=True):
    print "value:", event.value, "modified:", event.modifiedIndex, "created:", event.createdIndex

2) set keys will be receive event

etcdctl set /hello/test value1

3) open terminal set other keys

for i in $(seq 1 1 1010)
do
    etcdctl set /world/test value
done

4) find the client connect which etcd node

netstat -an | grep ES | grep 10.5.117*

5) go to etcd node and kill it

ssh 10.5.117.*
kill -9 (etcd-pid)

6) client reconnect and cause error

Traceback (most recent call last):
... ...
etcd.EtcdEventIndexCleared: The event in requested index is outdated and cleared : the requested history has been cleared [6873044/6872887]
lavagetto commented 7 years ago

I don't like masking a real issue (you effectively lose those 1000 events AIUI) with some code. I don't think we should merge this; also I'm not sure what that "if index" accomplishes.

lavagetto commented 7 years ago

to be clear, I do believe there is an issue in the code there, but should be resolved differently and properly tested.