pinterest / pymemcache

A comprehensive, fast, pure-Python memcached client.
https://pymemcache.readthedocs.io/
Apache License 2.0
768 stars 177 forks source link

Unicode character in key not detected if python2 str #117

Closed nichochar closed 7 years ago

nichochar commented 7 years ago

User tried to use key 'rpgv:https://www.pacificsales.com/brands/viking?ref=166_725&loc=soc_\xb1901520_%c3%abuy!' and no MemcacheIllegalInputError was raised, which causes obsure bug with stack trace below

Traceback (most recent call last):
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/services/utils/task_flow.py", line 165, in _run
return self.run()
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/services/utils/task_flow.py", line 160, in run
return self.f(*self.args, **self.kwargs)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/common/utils/decorators.py", line 556, in __get__
value = self.func(obj)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/api/serializers/contexts.py", line 1040, in rich_summary_map
return core.RichPinGridData.manager.get_many(ids)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/core/managers/rich_pin/rich_pin_view_manager.py", line 143, in get_many
return _get_many(model_ids)
File "/mnt/virtualenv/local/lib/python2.7/site-packages/pinstatsd/statsd.py", line 77, in decorated_function
return f(*args, **kwargs)
File "/mnt/virtualenv/local/lib/python2.7/site-packages/pinstatsd/statsd.py", line 98, in decorated_function
return_value = func(*args, **kwargs)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/core/managers/rich_pin/rich_pin_view_manager.py", line 138, in _get_many
models = super(FilteredRichPinViewManager, self).get_many(model_ids)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/core/managers/hbase_managers.py", line 572, in get_many
many_as_dicts = self.get_many_as_dicts(model_ids)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/core/managers/hbase_managers.py", line 552, in get_many_as_dicts
fetched_dicts = _get_many_as_dicts(self, model_ids)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/data_clients/mc_objects.py", line 154, in fn
lzero_missed_keys) or {}
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/data_clients/memcache.py", line 222, in wrap_f
return f(self, *args, **kwargs)
File "/mnt/builds/qB9LY9wQQXa4xZTJHZEv8A_d2eb341/data_clients/memcache.py", line 451, in get_many
result = mc.get_many(keys)
File "/mnt/virtualenv/local/lib/python2.7/site-packages/pymemcache/client/base.py", line 407, in get_many
return self._fetch_cmd(b'get', keys, False)
File "/mnt/virtualenv/local/lib/python2.7/site-packages/pymemcache/client/base.py", line 689, in _fetch_cmd
key = checked_keys[key]
KeyError: 'rpgv:https://www.pacificsales.com/brands/viking?ref=166_725&loc=soc_\xb1901520_%c3%abuy!'
nichochar commented 7 years ago

I think what's happening is that python2 is reporting that key as a str, which means it's not getting through the clause

    if isinstance(key, six.text_type):
        try:
            key = key.encode('ascii')
        except UnicodeEncodeError:
            raise MemcacheIllegalInputError("Non-ASCII key: '%r'" % (key,))

What is going on is that python2 str is not caught in {{six.text_type}} ([here is the documentation|https://pythonhosted.org/six/#six.text_type] so the check doesn't happen. (we weren't the only ones

nichochar commented 7 years ago

Going to write a PR to fix this.