bmuller / txyam

Yet Another Memcached (YAM) client for Python Twisted
Other
10 stars 5 forks source link

getMultiple not wrapped correctly? #11

Open Jasim9 opened 7 years ago

Jasim9 commented 7 years ago

I am trying the following code for getMultiple:

    p=0.03
    keys=[]
    for i in range(0,10):
        size=(np.random.geometric(p)%100)+1
        indx=random.randint(0,6000)
        keys.append("%05d%06d"%(size,indx))
    print keys
    mc.getMultiple(keys).addCallbacks(callback,errback)

Gives the following:

['00027003734', '00025003277', '00013003452', '00014000903', '00014002815', '00008004336', '00021000633', '00002005591', '00038000415', '00003002586']
Unhandled Error
Traceback (most recent call last):
  File "asyncsimLoad.py", line 46, in <module>
    main()
  File "asyncsimLoad.py", line 22, in main
    reactor.run()
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1195, in run
    self.mainLoop()
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 1204, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/base.py", line 825, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "asyncsimLoad.py", line 41, in test
    mc.getMultiple(keys).addCallbacks(callback,errback)
  File "/usr/local/lib/python2.7/dist-packages/txyam/client.py", line 31, in wrapper
    func = getattr(self.getClient(key), cmd)
  File "/usr/local/lib/python2.7/dist-packages/txyam/client.py", line 59, in getClient
    return self.hash_ring.get_node(key)
  File "/usr/local/lib/python2.7/dist-packages/hash_ring/hash_ring.py", line 94, in get_node
    pos = self.get_node_pos(string_key)
  File "/usr/local/lib/python2.7/dist-packages/hash_ring/hash_ring.py", line 108, in get_node_pos
    key = self.gen_key(string_key)
  File "/usr/local/lib/python2.7/dist-packages/hash_ring/hash_ring.py", line 154, in gen_key
    b_key = self._hash_digest(key)
  File "/usr/local/lib/python2.7/dist-packages/hash_ring/hash_ring.py", line 165, in _hash_digest
    m.update(key)
exceptions.TypeError: must be string or buffer, not list

And the wrap command in txyam does not account for a list of keys being passed in:

def wrap(cmd):
    """
    Used to wrap all of the memcache methods (get,set,getMultiple,etc).
    """
    def wrapper(self, key, *args, **kwargs):
        func = getattr(self.getClient(key), cmd)
        return func(key, *args, **kwargs)
    return wrapper

hence the error, is there a workaround for this?

bmuller commented 7 years ago

Due to the consistent hashing algorithm being used, the keys would have to be separated into groups for each client, and then the clients with matching keys would then be queried. This is definitely a bug - I'd be happy to look at a PR but am not likely going to be able to look at this for quite a while.