search

workflo / shaback

A hash digest based backup tool
GNU General Public License v3.0
4 stars 0 forks source link

restore (?) leaks memory #8

Closed raimi closed 11 years ago

raimi commented 11 years ago

I had to recover my root filesystem today. Some 90GB on a little Qnap NAS on Fast Ethernet. Recover speed was about 20GB/hour which is not that great but it is limited by seek times on the NAS (better fs, better hardware required).

Anyway, major bug: shaback consumes memory during restore. 8GB of RAM were not enough - it went out of mem at about 80GB. Here is what that looked like:

[...] [E] Cannot restore file ./usr/share/doc/cl-asdf/asdf/Components.html: inflateInit failed: insufficient memory [f] ./usr/share/doc/cl-asdf/asdf/Using-ASDF.html [E] Cannot restore file ./usr/share/doc/cl-asdf/asdf/Using-ASDF.html: inflateInit failed: insufficient memory [f] ./usr/share/doc/cl-asdf/asdf/operate.html [E] Cannot restore file ./usr/share/doc/cl-asdf/asdf/operate.html: inflateInit failed: insufficient memory [f] ./usr/share/doc/cl-asdf/asdf/Configuring-ASDF.html [E] Cannot restore file ./usr/share/doc/cl-asdf/asdf/Configuring-ASDF.html: inflateInit failed: insufficient memory [f] ./usr/share/doc/cl-asdf/asdf/TODO-list.html [E] Cannot restore file ./usr/share/doc/cl-asdf/asdf/TODO-list.html: inflateInit failed: insufficient memory [f] ./usr/share/doc/cl-asdf/copyright [E] Cannot restore file ./usr/share/doc/cl-asdf/copyright: inflateInit failed: insufficient memory [f] ./usr/share/doc/cl-asdf/README [E] Cannot restore file ./usr/share/doc/cl-asdf/README: inflateInit failed: insufficient memory [f] ./usr/share/doc/cl-asdf/README.Debian [E] Cannot restore file ./usr/share/doc/cl-asdf/README.Debian: inflateInit failed: insufficient memory [f] ./usr/share/doc/cl-asdf/asdf.pdf.gz [E] Cannot restore file ./usr/share/doc/cl-asdf/asdf.pdf.gz: inflateInit failed: insufficient memory [f] ./usr/share/doc/cl-asdf/changelog.Debian.gz [E] Cannot restore file ./usr/share/doc/cl-asdf/changelog.Debian.gz: inflateInit failed: insufficient memory [d] ./usr/share/doc/xorg inflateInit failed: insufficient memory

real 188m1.609s user 11m58.593s sys 6m37.237s

I had to continue manually from there (recovering the remaining 5 directories of /).

workflo commented 11 years ago

Maybe the read-cache file got too big and flooded your RAM disk?

raimi commented 11 years ago

I don't know. Both the "VIRT" and "RES" columns of top grew slowly but constantly. Didn't check that file on disk.

workflo commented 11 years ago

Have to take a deeper look. Maybe some mem leaks in calling inflate and friends...

workflo commented 11 years ago

Seems like a didn't close inflate/deflate contexts correctly/at all.

Should be fixed now!

workflo commented 11 years ago

According to my "top" the issue seems to be fixed :)