search

diegows / wanproxy

My Wanproxy repo - WAN Accelerator using dictionary and gzip compression.
http://www.wanproxy.org/
Other
12 stars 3 forks source link

XCodecEncoder::encode() is calling XCodecCache->lookup() too many times #6

Closed diegows closed 11 years ago

diegows commented 12 years ago

There is a bug in the encoder function, that makes XCodecEncoder::encode() function to call cache lookup() thousand times. for each segment to encode.

diegows commented 11 years ago

This is not a bug.

From Juli (original author of Wanproxy):

There's a speed/size trade-off in deduplication. If the xcodec segment size is 2048, imagine that you have a 2049 byte file come across the network. First, the codec checks at offset 0 to see if there is a segment that has been seen before. If there is not, it then checks at offset 1. The cache lookups need to be fast for this to be worthwhile, obviously. The reason this is necessary is that if the data comes to the encoder aligned differently than it was before, for whatever reason (and there are many reasons), then it needs to be able to scan ahead to find a segment it has seen before, so it can then catch up and continue deduplicating.