[-] Error (Module shivapushtodb.py) - insertSpam (1064, "You have an error in your SQL syntax

[genericcx]

Hi Guys,

Just trying out the HP, and I like it :) Nice job so far . Keep up the good work. I get a couple of errors every now and again, so thought I would put them here, in case its something not seen yet.

{{{ Error Message: [-] Error (Module shivapushtodb.py) - insertSpam (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 's@yahoo.com', '\n\xe5\x8f\xaf\xe6\x84\x9b\xe7\xb3\xbb\xe7\xbe\x8e\xe5\xa5\xb3\xe5\x88\x9d\xe7\x99\xbb\xe5\xa0\xb4\xe5\xb0\x87\xe5\xa5\xb9\xe7\xac\xac\xe4\xb8\x80\xe9\x83\xa8av\xe7\x8d\xbb\xe7\xb5\xa6\xe4\xbd\xa0\xe5\x80\x91!\n\n\xe5\x8f\xaf\xe6\x84\x9b' at line 1")you shall find sample in distorted directory }}}

Let me know if you need further information

[b0nd]

Hi cucx, thanks for reporting the bug. Nearby above stated error in log file you shall see unique id of spam sample that got error. Great if you could pass me that sample. Check "distorted" directory for it. Path to it shall be in your shiva.conf file.

Regards

[genericcx]

hi bond, thanks for the fast reply. Strangely it hasn't made it's way in there. I've only relic messages from several days ago there, and none of those have any ref to yahoo {{{ grep -Rl yahoo distorted/ (shivaAnalyzer)root@removed:/removed/removed/shiva/shiva# }}}

Maybe it cannot decode the Chinese properly?

{{{

print "\n\xe5\x8f\xaf\xe6\x84\x9b\xe7\xb3\xbb\xe7\xbe\x8e\xe5\xa5\xb3\xe5\x88\x9d\xe7\x99\xbb\xe5\xa0\xb4\xe5\xb0\x87\xe5\xa5\xb9\xe7\xac\xac\xe4\xb8\x80\xe9\x83\xa8av\xe7\x8d\xbb\xe7\xb5\xa6\xe4\xbd\xa0\xe5\x80\x91!\n\n\xe5\x8f\xaf\xe6\x84\x9b".decode("utf8","replace")

可愛系美女初登場將她第一部av獻給你們!

可愛 }}}

heres ones that did make it in the DB fine. They would be pretty similar I imagine:

{{{ id: b9e61b86448ab36c52d0836c23e40786 from: depart@yahoo.com subject: 超美形好色妹誘惑-各種地方做愛-口交-無套內射 to: goman226@yahoo.com.tw,goman2842@yahoo.com.tw,gomap.tw@yahoo.com.tw,gomandyhotmail@yahoo.com.tw,gomaluko@yahoo.com.tw,gomama1798@yahoo.com.tw,gomamon.tw@yahoo.com.tw,goman226@yaeahoo.com.tw,goman224@hotmail.com.tw,gomarlan@yahoo.com.tw,gomana3604@yahoo.com.tw,gomarryonepiece@yahoo.com.tw,gomark1111@yahoo.com.tw,gomasaken@gmail.com,goman178@yahoo.com.tw,gomako123@yahoo.com.tw,gomanto123@pchome.com.tw,gomanba9009417@yahoo.com.tw,gomama6569@yahoo.com.tw,goman224@yahoo.com.tw,gomasaino@yahoo.com.tw,goman226@yahoo.com.tq,gomangono1@yahoo.com.tw,gomaritakimo@yahoo.com.tw,gomanbobo@yahoo.com.tw,goman1116@yahoo.com.tw,gomamon_555@yahoo.com.tw,goman1112000@yahoo.com.tw,gomamama2001@yahoo.com.tw,gomanalong@yahoo.com.tw,goman180@yahoo.com.tw textMessage: 超美形好色妹誘惑-各種地方做愛-口交-無套內射

超美形好色妹誘惑-各種地方做愛-口交-無套內射

http://ppt.cc/rsIv

htmlMessage: totalCounter: 31 ssdeep: 3:+X7vno4xQUGdrqWmdW5HbU7vno4xQUGdrqWmdW3NViOVUPt47vno4xQUGdrqWmd0:+7o2GMWOIwo2GMWOyCOVvo2GMWO0 headers: (MIME-Version, 1.0) (Date, Thu, 16 Jan 2014 22:58:03 +0800) (From, "=?utf-8?b?5a6F55S35omT6Zu75YuV77yM5aeQ5aeQ5oCn5oW+5oOz5YGa5oSb77yM5LiA55u0?= =?utf-8?b?5Yu+5byV5LuW5LiK5bqK5bm55aW5?=" depart@yahoo.com) (Subject, =?utf-8?b?IOi2hee+juW9ouWlveiJsuWmueiqmOaDkS3lkITnqK7lnLDmlrnlgZrmhJst5Y+j?= =?utf-8?b?5LqkLeeEoeWll+WFp+WwhA==?=) (To, goman226@yahoo.com.tw) (X-Mailer, Microsoft Outlook Express 6.00.2800.1106) (X-Mimeole, Produced By Microsoft MimeOLE V6.00.2800.1106) (X-Msmail-Priority, Normal) (X-Priority, 3) (Content-Type, text/plain; charset="utf-8") (Content-Transfer-Encoding, base64)

  length: 151

1 row in set (0.00 sec)

}}}

[b0nd]

Thanks! Sadly cannot dive deep without faulty sample but language shall not be an issue; unicode handling is there in code and these Taiwanese samples have helped a lot in improving code in past :) Btw, please use 'development' branch, which is couple of commits ahead of 'master'. I'll bring master branch to a stable state soon.

[genericcx]

HI,

Ok, great thanks. I swiched to dev branch now, i still get some of the some errors, but they don't seem to be saving them in the distorted folder. Will keep you posted when i work out why they are not doing that.

[genericcx]

It's strange I'm getting still quite a few of these, however none of them are making its way to the distorted folder.

The folder must be working as I get messages in there when a different error occurs like

{{{ " Error (Module shivamailparser.py) - some issue in writePartsRecurse function" }}}

and these messages do correctly make their way into the distorted folder.

[b0nd]

Yeah, just cross verified the code. Any error confronted in shivapushtodb.py module doesn't move bad sample to 'distorted' folder. I need to figure out how that could be done as this module doesn't work directly on files (spams) but on data which is in memory and has already been processed by previous modules in chain. Surely it can be done, just matter of spending quality time on it. I'll keep this ticket open.

Thanks!

[b0nd]

Hi cucx, Could you please share your e-mail id with me? Together we can work on getting better data on your sensor.

Cheers!