Improve "Generate List Cache" performance

[GoogleCodeExporter]

Currently, it can take a LONG time for the program to "Generate List
Cache", even after doing something minor like adding a single IP to your
permanent block/allow list.  And this blocks your use of the program in the
meantime.

A few thoughts to consider:

1.  When generating list-cache, do it to a new file as a background task,
then "jumpswitch" over to it once it's ready.  This should minimize the
amount of time for which the program is unresponsive.

2.  Look into ways we could perform a "light list-cache update", for the
example case of when the user has only added a single IP address to one of
his personal lists.

Might be worth separating this out into two bugs, once we start to look at
it in earnest.  Definitely not gonna happen before v1.0.

Original issue reported on code.google.com by peerbloc...@gmail.com on 24 Jul 2009 at 6:05

[GoogleCodeExporter]

Original comment by peerbloc...@gmail.com on 24 Jul 2009 at 6:12

• Changed title: Improve "Generate List Cache" performance

[GoogleCodeExporter]

Additional discussion of these issues can be found in this forum thread:
(http://forums.peerblock.com/read.php?3,944).

Original comment by peerbloc...@gmail.com on 24 Sep 2009 at 4:20

[GoogleCodeExporter]

Removing 'After1.0' release-targetting.

Original comment by peerbloc...@gmail.com on 29 Sep 2009 at 3:58

• Removed labels: Milestone-After1.0

[GoogleCodeExporter]

As this is one of the most "popular" issues on the issue-tracker, targetting 
this to
be looked into for 1.1

Original comment by peerbloc...@gmail.com on 29 Sep 2009 at 4:04

• Added labels: Milestone-Release1.1

[GoogleCodeExporter]

Original comment by peerbloc...@gmail.com on 30 Sep 2009 at 4:20

• Added labels: Milestone-1.1
• Removed labels: Milestone-Release1.1

[GoogleCodeExporter]

Raising priority to high due to user interest.

Original comment by peerbloc...@gmail.com on 1 Oct 2009 at 5:12

• Added labels: Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Currently, here's the way List-Cache Generation works:

Generating List Cache (current)
1.  We take each of your Block Lists, combine them together into one "Master 
Block List"
2.  We take each of your Allow Lists, combine them together into one "Master 
Allow List"
3.  We start with this Master Block List, and take away all the ranges from the
Master Allow List
4.  We save the result as cache.p2b
5.  We tell the driver to start using the ranges from this newly-generated 
cache.p2b file

So when you right-click on something and select "Permanently Allow 1.2.3.4", we 
first
add this IP address to your ...\lists\permallow.p2b file, then run through 
steps 1-5
above.  This is slow, and annoying.

What we could consider doing instead is the following:

Generating List Cache (full) 
1.  Take each of your Block Lists, combine them together into one "Master Block 
List"
2.  Save the contents of the Master Block List to cache_block.p2b
3.  Take each of your Allow Lists, combine them together into one "Master Allow 
List"
4.  Save the contents of the Master Allow List to cache_allow.p2b
5.  Start with cache_block.p2b, and take away all the ranges from 
cache_allow.p2b
6.  Save the result as cache.p2b
7.  Tell the driver to start using the ranges from this newly-generated 
cache.p2b file

This would let us implement the following "lightweight" List Cache Generation 
algorithm:

Generating List Cache (light)
0.  Take newly-Allowed IP address, add it to cache_allow_new.p2b; take 
newly-Blocked
IP address, add it to cache_block_new.p2b
1.  Load cache_block.p2b, add ranges from cache_block_new.p2b, take away ranges 
from
cache_allow.p2b, then take away ranges from cache_allow_new.p2b
2.  Save the result as cache.p2b
3.  Tell the driver to start using the list of ranges from the new cache.p2b

This should be MUCH faster, since we won't need to regenerate the Master Block 
List
and Master Allow List each time you permanently allow an IP address.  Will cost 
a
small amount of disk-space - basically, we're doubling the amount of space used 
by
cache.p2b - but a few megabytes of disk-space would be a small price to pay I 
think.

The next time we download newly-updated lists from the Internet, we'll still 
need to
run through the "Full" Generating List Cache algorithm (and delete the two 
_new.p2b
files), but this should at least be less-painful while you're actively using the
program.  We'll need to address this part of the pain in a later patch, moving 
this
Generating List Cache stuff to a background thread so that the user-impact will 
be
minimal (though it will still be chewing up CPU in the background).

Original comment by peerbloc...@gmail.com on 2 Feb 2010 at 6:18

[GoogleCodeExporter]

Can multi-threading possibly be applied to some of the steps? 

Original comment by Praese...@gmail.com on 2 Feb 2010 at 7:36

[GoogleCodeExporter]

I appreciate that everything is laid out in step by step here as to exactly 
what is 
done. I know if I allow/block something for 15-min or an hour it does it almost 
instantaneously. Obviously as it is only temporary it isn't messing with the 
master 
list and doesn't take several seconds to rebuild like adding permanently does.

What I was thinking is you could have it "temporarily" add it using a similar 
method 
you have for the short duration allow/block and have it rebuild the list into 
the 
master list periodically (I donno maybe a set interval here like 5-min or 
15-min), 
and on program close. And make sure it only rebuilds if changes were made 
obviously. 
This way from a user standpoint, the IP address will be allowed/blocked almost 
instantaneously and the cache will only be rebuilt periodically as to not bog 
down 
the system.

I am glad that the multiple ranges was added because it was annoying to have to 
wait 
after adding each IP to the list but I feel this would further improve overall 
performance.

Original comment by dye...@gmail.com on 23 Feb 2010 at 2:05

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

1. Could you allow multiple selections to be made, e.g. Ctrl-click adds 
additional IP 
addresses to be actioned, and Ctrl-Shift-click adds a whole chunk of IP 
addresses to 
be actioned?

2. Could you allow IP ranges to be actioned, e.g. a context menu option for 
1.2.3.4 
might be:
Allow for 15 mins
Allow for 1 hour
Allow permanently
Allow 1.2.3.xxx for 15 mins
Allow 1.2.3.xxx for 1 hour
Allow 1.2.3.xxx permanently

Original comment by lamanga2...@hotmail.com on 13 May 2010 at 3:07