Closed GoogleCodeExporter closed 8 years ago
Am investigating now, you shouldn't be getting those, basically a chunk should
contain either: S or A headers, it discards them if for some reason the header
isn't either. Just waiting for the timeout to finish so I can pull up the error
data again.
Original comment by sam%beav...@gtempaccount.com
on 5 Jul 2010 at 6:59
Could you try the class here:
http://phpgsb.googlecode.com/svn/branches/debug/phpgsb.class.php
It should dump any chunkdata into .debug files
Could you reply with a few of those attached please?
Original comment by sam%beav...@gtempaccount.com
on 5 Jul 2010 at 7:49
I tried your class. You can access the debug files here:
http://testsite.tcpiputils.com/DUMP0.debug its till 17.
Original comment by i...@tcpiputils.com
on 5 Jul 2010 at 8:26
Original comment by sam%beav...@gtempaccount.com
on 5 Jul 2010 at 8:30
i see you are using the mb_ functions. Perhaps you need to know i have a
default encoding set to utf8.
Original comment by i...@tcpiputils.com
on 5 Jul 2010 at 8:31
So do I, I'll try processing some of those chunks on my server and I'll see if
it manages to process them.
Original comment by sam%beav...@gtempaccount.com
on 5 Jul 2010 at 8:33
Odd it processed that first one fine, in your listupdater.php can you find:
$phpgsb->runUpdate();
and replace with:
$chunkinfo = file_get_contents("DUMP0.debug");
print_r($phpgsb->processChunks($chunkinfo));
And let me know what it outputs?
Original comment by sam%beav...@gtempaccount.com
on 5 Jul 2010 at 9:03
you can see the output here:
http://testsite.tcpiputils.com/listupdater.php
Original comment by i...@tcpiputils.com
on 5 Jul 2010 at 9:15
Thats very odd, we're using the same code, the same debug file and yet we get
different outputs.. hmm (that error at the top of yours for some reason is
there because the reading of the data seems to go awry at the end of the first
chunk.)
I'll do some more tests tommorow and will likely have some more code for you to
try!
You could try removing the mb_ inside processchunks function and see if that
makes a difference. It'll help eliminate that as a source of the problem.
Original comment by sam%beav...@gtempaccount.com
on 5 Jul 2010 at 9:30
i changed the mb_ function and now i dont get any error:
http://testsite.tcpiputils.com/listupdater1.php
Original comment by i...@tcpiputils.com
on 5 Jul 2010 at 9:35
Hmm very odd, I originally used the functions as a precaution though it should
be okay with the regular substr, I'll get it replaced in the source. Thanks for
flagging this up! You might want to clear the database and start fresh now to
avoid any corruption. Is it all working correctly now?
Original comment by sam%beav...@gtempaccount.com
on 5 Jul 2010 at 9:44
[deleted comment]
yes i deleted the comment because i was not finished yet...
but i still get two false results with your lookup example:
http://testsite.tcpiputils.com/lookup.php
Original comment by i...@tcpiputils.com
on 5 Jul 2010 at 10:36
The database could take around 24 hours to fully populate the database (you
have to keep running the updater, I'd put it on a cron running every minute [it
won't actually update every minute but will ping the timeout file]). Google
don't provide all the information in one go you have to keep updating and
eventually you get the majority of the chunks.
Original comment by sam%beav...@gtempaccount.com
on 5 Jul 2010 at 10:41
There are still problems...only the *index tables get populated.
I did some debug and i see for example the Insert hostkeys index query is
failing with the error Field 'FullHash' doesn't have a default value.
try this one on your machine:
INSERT INTO `phpgsb`.`goog-malware-shavar-s-hosts` (`ID`,`Hostkey`,`Chunknum`,`Count`) VALUES ('','cfa8d53a','33961','00')
Original comment by i...@tcpiputils.com
on 6 Jul 2010 at 6:26
i see the problem..the field fullhash has the property Not Null and there is
not a default value. If i change it to a default value i get a new error:
Incorrect integer value: '' for column 'ID' at row 1
Original comment by i...@tcpiputils.com
on 6 Jul 2010 at 6:35
Could you try using this version:
http://phpgsb.googlecode.com/svn/trunk/phpgsb.class.php
And let me know the results.
Original comment by sam%beav...@gtempaccount.com
on 6 Jul 2010 at 7:42
the FullHash is solved..now i get
INSERTED goog-malware-shavar SUB HOST KEYS Incorrect integer value: '' for
column 'ID' at row 1...
Original comment by i...@tcpiputils.com
on 6 Jul 2010 at 7:57
That shouldn't be happening... can you check that the ID field is set to
Auto-Increment in the database?
Original comment by sam%beav...@gtempaccount.com
on 6 Jul 2010 at 8:01
yes it is AI. You have to remove the column ID from Your insert statement
because it is automaticly populated by mysql.
Original comment by i...@tcpiputils.com
on 6 Jul 2010 at 8:14
Hmm I've never had a problem with that before, I've removed the ID column in
this version:
http://phpgsb.googlecode.com/svn/trunk/phpgsb.class.php
Does that prevent the problem?
Original comment by sam%beav...@gtempaccount.com
on 6 Jul 2010 at 8:17
it seems to work now..i will perform some test to be sure..thanx
Original comment by i...@tcpiputils.com
on 6 Jul 2010 at 8:50
Could you confirm if its now working now?
Original comment by sam%beav...@gtempaccount.com
on 7 Jul 2010 at 4:00
yes it works but i get a lot and lot notices:
PHP Notice: Undefined index: PREFIX in phpgsb.class.php on line 307
Can You fix this?
Original comment by i...@tcpiputils.com
on 7 Jul 2010 at 6:02
For some reason you're setup must be using error_reporting(E_ALL);
In listupdater.php and lookup.php the following to the top (after <?php ):
error_reporting(E_ALL ^ E_NOTICE);
At a later date I'll rework the code to check for existances of variables more.
(They're not actually errors)
Original comment by sam%beav...@gtempaccount.com
on 7 Jul 2010 at 6:06
Original comment by sam%beav...@gtempaccount.com
on 7 Jul 2010 at 7:44
Original issue reported on code.google.com by
i...@tcpiputils.com
on 2 Jul 2010 at 10:40