storing large objects

[jeroen]

rredis seems to be ok for small sized, data, but when I try to store a large object the connection seems to die:

> redisSet("test", rnorm(1e8))
Error in doTryCatch(return(expr), name, parentenv, handler) : 
  ERR Protocol error: invalid bulk length
In addition: Warning messages:
1: In writeBin(v, con) : problem writing to connection
2: In writeBin(.raw("\r\n"), con) : problem writing to connection

Any idea what is going on?

[bwlewis]

This is new to me, thanks for pointing it out.

The limit you've found is within redis itself. I'm checking now to pin down the exact limit and whether or not it's uniform. it appears to be hard-coded in the redis source.

-bryan On Jul 15, 2012 11:20 AM, "jeroenooms" < reply@reply.github.com> wrote:

rredis seems to be ok for small sized, data, but when I try to store a large object the connection seems to die:

    > redisSet("test", rnorm(1e8))
    Error in doTryCatch(return(expr), name, parentenv, handler) :
      ERR Protocol error: invalid bulk length
    In addition: Warning messages:
    1: In writeBin(v, con) : problem writing to connection
    2: In writeBin(.raw("\r\n"), con) : problem writing to connection

Any idea what is going on?

---

Reply to this email directly or view it on GitHub: https://github.com/bwlewis/rredis/issues/6

[userbc]

I am having the same problem as jeroenooms. Did you ever manage to find a solution to this or is it still unsolved?

Thanks

[bwlewis]

Hi! Redis values are limited to 512MB, see for example:

http://redis.io/topics/data-types

Any R object exceeding this length will result in an invalid bulk length error.

There was a bug in Redis affecting HMSET that could result in this kind of error happening randomly. it was fixed months ago. Here is a reference: https://github.com/antirez/redis/issues/673

The upshot is, if you need to store large objects in redis, you'll have to compile a custom redis server.

[userbc]

Thank you. You say that to store objects larger than 512MB I need to compile a custom redis server, is this something I can change in the redis.conf file? Can you point me in the direction of what I need to change please?

[bwlewis]

This is actually an open bug in Redis. Unfortunately, it's not a configuration option. And it appears that even changing the source code of Redis to support this is trickier than I thought. See the following bug thread for a discussion on this topic:

https://github.com/antirez/redis/issues/757

So it looks like 512MB will be the limit for a while yet.