bwlewis / rredis

R client for Redis
http://illposed.net/
93 stars 25 forks source link

Question: R object size limitation of 512 MB #27

Closed jaehyeon-kim closed 9 years ago

jaehyeon-kim commented 9 years ago

Hi

Just a quick question. In the vignette, the size limitation of a R object is indicated below.

When I checked its data types, however, the maximum number of elements of lists, sets and hashes is 2^32 - 1 (http://redis.io/topics/data-types) while the maximum size of its key and value is 512 MB.

So I guess, for example, if a R vector is created, as long as each value (and the key) doesn't exceed 512 MB, it'd be possible to add.

Please inform if the guess is incorrect.

Regards Jaehyeon

bwlewis commented 9 years ago

Indeed redis sets and lists have an indexing limit of 2^32-1.

You are free to store each element of an R vector in a redis list or set, and then you can store 2^32-1 R objects in a single list or set. But note that these are stored as separate R objects, not elements of a single R vector.

The default behavior of the rredis package serializes R objects (whatever they may be) and stores them as redis "strings" (binary blobs). Using this mechanism, 512MB is the redis-imposed limit of any single R object, vector or otherwise. Again, you are free to use a different convention to get around this redis limit.