redis backend implementation

[nandub]

A first stab at creating a redis backend, please review and let me know.

[nandub]

Two of the test fails with

doctestall -l dbcps.backends.rediscps
Testing dbcps.backends.rediscps
**********************************************************************
File "/Users/fortiz/development/pythondev/python-libcps/dbcps/backends/rediscps.py", line 13, in dbcps.backends.rediscps.Redis
Failed example:
    s['hello']
Expected:
    'world'
Got:
    RawData(adbcd29fbccfbc838cbe8c8f8c928c8c8b9484)
**********************************************************************
File "/Users/fortiz/development/pythondev/python-libcps/dbcps/backends/rediscps.py", line 25, in dbcps.backends.rediscps.Redis
Failed example:
    s['blueberry']
Expected:
    'pancakes'
Got:
    RawData(adbcd29fbccfbc838c908b918c8e8b938b918c8b8bbc8c9384)
**********************************************************************
1 items had failures:
   2 of  13 in dbcps.backends.rediscps.Redis
***Test Failed*** 2 failures.
2 failures, 13 tests.

[nandub]

I tried to convert the RawData to String and I get a utf-8 decode error

Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dbcps import storage
>>> backend = ('rediscps', ['rotate', -91])  
>>> s = storage.Storage([backend])
>>> s['hello'] = 'world' 
>>> s['hello']
RawData(adbcd29fbccfbc838cbe8c8f8c928c8c8b9484)
>>> s['hello'].toString()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/ejtp/util/py2and3.py", line 222, in toString
    return String(self.export().decode('utf-8'))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xad in position 0: invalid start byte

Not sure if I'm missing something. If I create a RawData manually I get a different outcome

>>> from ejtp.util import py2and3
>>> py2and3.RawData('world')
RawData(776f726c64)
>>> py2and3.RawData('world').toString()
String('world')
>>> py2and3.RawData('world')._data
(119, 111, 114, 108, 100)
>>> py2and3.RawData('world').toString()._data
u'world'

[nandub]

I merged the branch with your fixes, now the tests pass. Thanks. I think this pull request is ready to merge to master.

[MaddieM4]

This looks great! I'm gonna test it out with Redis tonight, see if it works on my machine - and if it does, I'll definitely merge it in. If all goes according to plan, this'll be merged on the first, in line with the stable-ization of EJTP v0.9.2, but that doesn't mean we have to wait 'till the last minute to confirm it works, etc.

[MaddieM4]

Confirmed working. I have one change I'd like to request, and then we're done.

The backend module should simply be called "redis". This doesn't cause a name conflict, because backends are never used in relative imports, and it seems like good practice to not add a superfluous "cps" to every backend name when they're going to be imported as:

 import dbcps.backends.appengine

or, in the event of a naming conflict in the importing code,

 from dbcps.backends import appengine as appengine_cps

But the truth is even those examples are stretches for realistic third-party code, since you'd really be instantiating with the Storage class anyways, which always does absolute imports internally. So it makes sense to name cleanly.

[nandub]

On my first iteration I called the backend redis but changed it because I was having name conflict with redis-py (import redis) package, but maybe it was something I was doing wrong in the first place.

I will rename and test.

[nandub]

I renamed the backend to redis and now I'm getting the error I was getting before

Testing dbcps.backends.redis
**********************************************************************
File "/Library/Python/2.7/site-packages/dbcps/backends/redis.py", line 10, in dbcps.backends.redis.Redis
Failed example:
    s = storage.Storage([backend])
Exception raised:
    Traceback (most recent call last):
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1254, in __run
        compileflags, 1) in test.globs
      File "<doctest dbcps.backends.redis.Redis[2]>", line 1, in <module>
        s = storage.Storage([backend])
      File "/Library/Python/2.7/site-packages/dbcps/storage.py", line 6, in __init__
        self.install(backends)
      File "/Library/Python/2.7/site-packages/dbcps/storage.py", line 10, in install
        dbh = make_dbh(*backend)
      File "/Library/Python/2.7/site-packages/dbcps/dbh.py", line 50, in make_dbh
        return dbh_module.dbh_class(*args)
      File "/Library/Python/2.7/site-packages/dbcps/backends/redis.py", line 30, in __init__
        self.db = redis.Redis(host, port)
      File "/Library/Python/2.7/site-packages/dbcps/backends/redis.py", line 29, in __init__
        DBH.__init__(self, handle, encryptor)
      File "/Library/Python/2.7/site-packages/dbcps/dbh.py", line 10, in __init__
        self.encryptor = crypto.make(encryptor)
      File "/Library/Python/2.7/site-packages/ejtp/util/py2and3.py", line 471, in __call__
        return self._decoratedFunc(self._func, *((func,)+args), **kwargs)
      File "/Library/Python/2.7/site-packages/ejtp/util/py2and3.py", line 607, in _decoratedFunc
        return func(*args, **kwargs)
      File "/Library/Python/2.7/site-packages/ejtp/crypto/encryptor.py", line 123, in make
        data = json.loads(data.export())
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 326, in loads
        return _default_decoder.decode(s)
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 360, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 378, in raw_decode
        raise ValueError("No JSON object could be decoded")
    ValueError: No JSON object could be decoded
**********************************************************************

Any thoughts?

[MaddieM4]

It's because you import the Python redis library under that name. Try this:

import redis as redislib
...
self.db = redislib.Redis(blah blah blah)

Basically any alternate name will do... we'll probably have some sort of best practice for that naming, but for now, do whatever. Let me know if that works, I'm on my phone so nothing is convenient except Twitter and texting, neither of which has much bearing on this issue. On Jan 31, 2013 7:16 PM, "Fernando Ortiz" notifications@github.com wrote:

I renamed the backend to redis and now I'm getting the error I was getting before

Testing dbcps.backends.redis

---

File "/Library/Python/2.7/site-packages/dbcps/backends/redis.py", line 10, in dbcps.backends.redis.Redis Failed example: s = storage.Storage([backend]) Exception raised: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1254, in run compileflags, 1) in test.globs File "<doctest dbcps.backends.redis.Redis[2]>", line 1, in s = storage.Storage([backend]) File "/Library/Python/2.7/site-packages/dbcps/storage.py", line 6, in __init self.install(backends) File "/Library/Python/2.7/site-packages/dbcps/storage.py", line 10, in install dbh = make_dbh(_backend) File "/Library/Python/2.7/site-packages/dbcps/dbh.py", line 50, in make_dbh return dbh_module.dbh_class(_args) File "/Library/Python/2.7/site-packages/dbcps/backends/redis.py", line 30, in init self.db = redis.Redis(host, port) File "/Library/Python/2.7/site-packages/dbcps/backends/redis.py", line 29, in init DBH.init(self, handle, encryptor) File "/Library/Python/2.7/site-packages/dbcps/dbh.py", line 10, in init self.encryptor = crypto.make(encryptor) File "/Library/Python/2.7/site-packages/ejtp/util/py2and3.py", line 471, in call return self._decoratedFunc(self.func, ((func,)+args), _kwargs) File "/Library/Python/2.7/site-packages/ejtp/util/py2and3.py", line 607, in _decoratedFunc return func(_args, _kwargs) File "/Library/Python/2.7/site-packages/ejtp/crypto/encryptor.py", line 123, in make data = json.loads(data.export()) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/init.py", line 326, in loads return _default_decoder.decode(s) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 360, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 378, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded

---

Any thoughts?

— Reply to this email directly or view it on GitHubhttps://github.com/campadrenalin/python-libcps/pull/2#issuecomment-12978872.

[nandub]

I did as you suggested but no matter how I do it still fails with the same error as above. Do you have any other preference of naming the backends to avoid conflicts like this?

[nandub]

Renamed back to rediscps now it works as expected.

[MaddieM4]

Finally figured it out, after spending my own time hacking on it. Change your import line to this:

redislib = __import__("redis", {})

This is the horrible, ugly absolute import syntax (one of the things that makes Python 3 more attractive to me by the day). This gets you exactly what you need. Then you change the self.db line to use redislib, and find/replace "rediscps" to "redis", and you're good.

[nandub]

Just changed the import signature and now everything works as expected. It looks ugly but it works :+1:

[MaddieM4]

This looks great. Merged.