phpgsb requires a lot of memory to process

[GoogleCodeExporter]

I wasn't able to update phpgsb without letting PHP use 64MB of memory.  This 
seems quite high.  The error always triggered in processChunks(), which makes 
sense.

That's a lot of memory for updating things.  Really phpgsb should write to disk 
(mySQL in this case) much sooner so that it doesn't need to consume so much 
memory.

Original issue reported on code.google.com by raccettu on 8 Nov 2010 at 3:52

[GoogleCodeExporter]

That's a very valid point, I'll have to go over the documentation etc to make 
sure thats possible as I know it has to process the entire chunk before it does 
any writes to make sure its all valid (or I'll have to put in an "undo" feature 
incase of any errors later on down the line). Hopefully this'll be on the next 
version.

Original comment by sam%beav...@gtempaccount.com on 18 Nov 2010 at 4:06

• Changed state: Accepted

[GoogleCodeExporter]

Perhaps mySQL transaction?

Original comment by raccettu on 18 Nov 2010 at 5:00

[GoogleCodeExporter]

Ahh I completely forgot about that! Very good idea, I'm very busy this week but 
hopefully I can start work on a newer version next week (Unless of course you'd 
be willing to contribute some code?); I hope to incorporate some other changes 
and simplifications that I've discovered since the last stable-ish release.

Original comment by sam%beav...@gtempaccount.com on 18 Nov 2010 at 5:17

[GoogleCodeExporter]

Original comment by sam%beav...@gtempaccount.com on 18 Nov 2010 at 5:24

• Added labels: Performance

[GoogleCodeExporter]

Search faster by INNODB and add `Hostkey` to index. It's good for me.

Original comment by artiy...@gmail.com on 25 Jan 2011 at 8:42

[GoogleCodeExporter]

That wouldn't actually fix this bug.  The problem is here is memory, not DB 
performance.

Original comment by raccettu on 25 Jan 2011 at 2:43

[GoogleCodeExporter]

Anyway, creating index speeds up things alot and reduces CPU impact from 
searches. Just add this to install.php:

create index hostkey on `googpub-phish-shavar-a-hosts` (Hostkey);
create index hostkey on `goog-malware-shavar-a-hosts` (Hostkey);

Original comment by cro...@gmail.com on 25 Feb 2011 at 11:00

[GoogleCodeExporter]

I've made some major changes to the class. Summary of changes:
-> Script will now update database at the end of each chunk. (Rather than at 
the end of each file it downloads).
-> Engine has been switched to InnoDB to support MySQL Transaction (enabling 
rollback incase of failure in response).
-> An index has been added on "Hostkey" in the hosts tables.

From benchmarking on updates memory usage has been reduced from ~35MB to ~5MB. 
Lookup speed has also been improved thanks to the new indexes (haven't got an 
exact figure for that).

I wrote a small upgrade script to help anyone upgrade from 0.1.* to the new 
version (0.2). It essentially just changes the engine type and adds the indexs, 
it takes a  minute or so to run just while MySQL changes the engine. The only 
file really changed is phpgsb.class.php, the other files remain the same.

I've added the download to the Downloads page (haven't made it featured yet 
while I'm still testing). Let me know how it runs or if you have any 
questions/problems!

Original comment by sam%beav...@gtempaccount.com on 26 Feb 2011 at 5:41

• Changed state: Awaiting

[GoogleCodeExporter]

Original comment by sam%beav...@gtempaccount.com on 13 Mar 2011 at 12:51

• Changed state: Fixed