search

rolffokkens / bdsync

a fast block device synchronizing tool
GNU General Public License v2.0
109 stars 26 forks source link

Memory consumption growing during operation #10

Closed sumpfralle closed 8 years ago

sumpfralle commented 8 years ago

Hi,

I recently noticed that the diffing process for my 375 GB volume causes the memory consumption of bdsync to grow up to about 2 GB. As far as I unerstand bdsync's operation I would assume that its memory usage is independent of the size of the volume to be processed. Thus I could imagine that some memory is not freed where it should be. The memory consumption of the server process (on the target host) is not growing.

Besides this small detail bdsync is doing its job perfectly fine. Thank you!

Cheers, Lars

rolffokkens commented 8 years ago

Seems somewhat memory-leaky indeed. Will look into this.

rolffokkens commented 8 years ago

Hi Lars,

Ik found a few memleaks. mtrace now assures me that these are gone. Could you give it a try?

Thanks,

Rolf

rolffokkens commented 8 years ago

Apart from the true memleaks, there's something else going on: bdsync uses extensive buffering of traffic between client and server to avoid any kind of congestion. A side effect of the current implementation is that this buffering results in excessive memory consumption when a lot of data is exchanged between client and server (speficically when --remdata is used). I'm working on a solution by earlier flushing data.

rolffokkens commented 8 years ago

Memory usage seems to be under control now completely. From start to finish (of a long bdsync) the total memory footprint stays < 20MB.

sumpfralle commented 8 years ago

Thank you! Now it runs even better ...