search

taoensso / carmine

Redis client + message queue for Clojure
https://www.taoensso.com/carmine
Eclipse Public License 1.0
1.15k stars 130 forks source link

Setting large bitfields and ignoring return values? #233

Closed theronic closed 4 years ago

theronic commented 4 years ago

When I try to set a billion bits on a bitfield, I get a "Java heap space" exception, but it works for fewer bits:

(wcar*
  (doseq [x (range 1e9)]
    (redis/bitfield "users:roles" "SET" "u1" x (mod x 2))
    (redis/bitfield "users:allowed?" "SET" "u1" x 1))
    (redis/bitcount "users:roles"))

How should I be committing these performantly and ignoring the intermediate write results? Do I need to batch them in separate calls to wcar? I know I can chain many bit ops in one call, but it's not clear what those limits are.

ptaoussanis commented 4 years ago

Hi there,

Everything in your doseq is being pipelined w/in a single wcar* call - and the results accumulated. You'll need to force the pipeline to end before it gets too big- either by calling wcar* multiple times (e.g. 1000x wcar*s each executing 1e6 ops), or by using the with-replies function.

Hope that helps!