memory usage problem

[awb99]

I am storing 2500 items via konserve:

(defn save [k item]
  (<!! (k/assoc-in store k item)))

(defn save-items [data-type data-key items]
  (info "storing to konserve ..")
  (doall
   (for [item items]
     (save [data-type (data-key item)] item)))
  [])

(defn store-xero-contacts-konserve [items]
  (k/save-items :xero-contact :ContactID items)

I call (store-xero-contats-konserve customers) to save 2500 contacts (each a map with customer contact information). The customers seq when spit as edn to a file is 2.5 MB.

I am getting out of memory exceptions when I do this. I tried to get it to work by setting :jvm-opts ["-Xmx32g"] This hack worked for the 2500 contacts. Then I did try to do the same for 4000 invoices, and this fails. It seems there is a max heap size my jvm can handle, and I am already at this level, so cannot increase it any more.

I have been using a different code, when I was using konserve to save only one item at a time. This never produced a memory problem.

So I guess the issue is that konserve releases the memory periodically. But in a big batch operation it does not have time to release memory.

What is bizyrre is that my data is only 2.5 MB. And heap in gigs region.

Any ideas?

[whilo]

Hey, I am sorry to hear that you have this memory issue. Do you use konserve with or without caching and which store backend do you use? Versions would also be good to know.

[whilo]

This is really mysterious to me. I am happy to help solve the issue, but I need more information as it is not reproducible like this. I will close it for now, but you can reopen with more information if the issue still persists.