Closed GoogleCodeExporter closed 8 years ago
That's a very valid point, I'll have to go over the documentation etc to make
sure thats possible as I know it has to process the entire chunk before it does
any writes to make sure its all valid (or I'll have to put in an "undo" feature
incase of any errors later on down the line). Hopefully this'll be on the next
version.
Original comment by sam%beav...@gtempaccount.com
on 18 Nov 2010 at 4:06
Perhaps mySQL transaction?
Original comment by raccettu
on 18 Nov 2010 at 5:00
Ahh I completely forgot about that! Very good idea, I'm very busy this week but
hopefully I can start work on a newer version next week (Unless of course you'd
be willing to contribute some code?); I hope to incorporate some other changes
and simplifications that I've discovered since the last stable-ish release.
Original comment by sam%beav...@gtempaccount.com
on 18 Nov 2010 at 5:17
Original comment by sam%beav...@gtempaccount.com
on 18 Nov 2010 at 5:24
Search faster by INNODB and add `Hostkey` to index. It's good for me.
Original comment by artiy...@gmail.com
on 25 Jan 2011 at 8:42
That wouldn't actually fix this bug. The problem is here is memory, not DB
performance.
Original comment by raccettu
on 25 Jan 2011 at 2:43
Anyway, creating index speeds up things alot and reduces CPU impact from
searches. Just add this to install.php:
create index hostkey on `googpub-phish-shavar-a-hosts` (Hostkey);
create index hostkey on `goog-malware-shavar-a-hosts` (Hostkey);
Original comment by cro...@gmail.com
on 25 Feb 2011 at 11:00
I've made some major changes to the class. Summary of changes:
-> Script will now update database at the end of each chunk. (Rather than at
the end of each file it downloads).
-> Engine has been switched to InnoDB to support MySQL Transaction (enabling
rollback incase of failure in response).
-> An index has been added on "Hostkey" in the hosts tables.
From benchmarking on updates memory usage has been reduced from ~35MB to ~5MB.
Lookup speed has also been improved thanks to the new indexes (haven't got an
exact figure for that).
I wrote a small upgrade script to help anyone upgrade from 0.1.* to the new
version (0.2). It essentially just changes the engine type and adds the indexs,
it takes a minute or so to run just while MySQL changes the engine. The only
file really changed is phpgsb.class.php, the other files remain the same.
I've added the download to the Downloads page (haven't made it featured yet
while I'm still testing). Let me know how it runs or if you have any
questions/problems!
Original comment by sam%beav...@gtempaccount.com
on 26 Feb 2011 at 5:41
Original comment by sam%beav...@gtempaccount.com
on 13 Mar 2011 at 12:51
Original issue reported on code.google.com by
raccettu
on 8 Nov 2010 at 3:52