Samples upload and analyze fine in CRITs Instance but source domain upload fails.

[markmaunu]

Hello - Both domain and sample submission were successfully submitting and that relationship added in the webstergd fork, but moving to the latest master samples but not domains now succeed. The first part of domain submission is the failure then it succeeds submitting and adding to CRITs.

Ex log below (obfuscated URL for safety and took out my legit MYAPIKEY ):

2015-04-04 17:02:20 139944445957952 Starting new HTTPS connection (1): 127.0.0.1 2015-04-04 17:02:20 139944445957952 Could not connect to CRITs when submitting domain kmlky.com 2015-04-04 17:02:20 139944445957952 Starting new HTTPS connection (1): 127.0.0.1 2015-04-04 17:02:23 139944445957952 "POST /api/v1/samples/ HTTP/1.1" 200 None 2015-04-04 17:02:23 139944445957952 Saved 927963665ed539335b740f704a3d546f to dump dir 2015-04-04 17:02:23 139944445957952 hxxp://healthygarcinialifestyle[.]com/wp-content/old/Paypal.co.uk/login/PP1/myaccount/147c1/home?cmd=_account-details&session=2439328f21281f85a23af8649ca47a4b&dispatch=6e0b2cd81e08706e9667626b79394a77be5598b0 hashes to 1bc3e8566b264866db13d93a8d5c446d 2015-04-04 17:02:23 139944445957952 Domain submission: https://127.0.0.1/api/v1/domains/|{'username': 'maltrieve', 'source': 'maltrieve', 'domain': u'healthygarcinialifestyle[.]com', 'api_key': 'MYAPIKEY'}

The version of CRITs I am running is: 4-master. Any logs etc., I can provide please let me know and thanks for this awesome tool !

[webstergd]

Hey Mark,

Sorry it is late and I am having trouble understanding your problem. Are you saying that domains are not added properly but that samples are?

On Sun, Apr 5, 2015 at 2:23 AM, markmaunu notifications@github.com wrote:

Hello - Both domain and sample submission were successfully submitting and that relationship added in the webstergd fork, but moving to the latest master samples but not domains now succeed. The first part of domain submission is the failure then it succeeds submitting and adding to CRITs.

Ex log below (obfuscated URL for safety and took out my legit MYAPIKEY ):

2015-04-04 17:02:20 139944445957952 Starting new HTTPS connection (1): 127.0.0.1 2015-04-04 17:02:20 139944445957952 Could not connect to CRITs when submitting domain kmlky.com 2015-04-04 17:02:20 139944445957952 Starting new HTTPS connection (1): 127.0.0.1 2015-04-04 17:02:23 139944445957952 "POST /api/v1/samples/ HTTP/1.1" 200 None 2015-04-04 17:02:23 139944445957952 Saved 927963665ed539335b740f704a3d546f to dump dir 2015-04-04 17:02:23 139944445957952 hxxp://healthygarcinialifestyle[.]com/wp-content/old/ Paypal.co.uk/login/PP1/myaccount/147c1/home?cmd=_account-details&session=2439328f21281f85a23af8649ca47a4b&dispatch=6e0b2cd81e08706e9667626b79394a77be5598b0 hashes to 1bc3e8566b264866db13d93a8d5c446d 2015-04-04 17:02:23 139944445957952 Domain submission: https://127.0.0.1/api/v1/domains/|{'username': 'maltrieve', 'source': 'maltrieve', 'domain': u'healthygarcinialifestyle[.]com', 'api_key': 'MYAPIKEY'}

The version of CRITs I am running is: 4-master. Any logs etc., I can provide please let me know and thanks for this awesome tool !

— Reply to this email directly or view it on GitHub.

[krmaxwell]

If you're just judging from the logs, don't. ;) See #149 for example. Our logging is not good right now.

So you've verified in CRITs that the domain is not there but the sample is?

[markmaunu]

Hey guys this is exactly the case the domain is not there but all of the samples get added.

[krmaxwell]

So the log indicates that the domain upload is succeeding (unless there's stuff after that from CRITs that we're not seeing in the issue). Have you verified that the user has access to the source, so that you're seeing all the data?

[markmaunu]

It looks like the last successful correlation I have is on 4-2-15 for a domain that was originally added and a sample pulled on 3-31-15.

flv3.dmrcdn[.]com 2015-04-02 05:46:49 maltrieve New dmrcdn[.]com 2015-04-02 05:46:45 maltrieve New thejsscripts[.]com 2015-04-02 05:46:34 maltrieve New www[.]solvusoft[.]com 2015-04-02 05:46:29 maltrieve New

Screenshot added for first domain listed above showing correlation

[krmaxwell]

What do the details look like if you go the other way, a sample that was just downloaded today?

[markmaunu]

Just verified that I have access to the source - added cnn.com successfully

[markmaunu]

Attached is a sample that uploaded today through maltrieve.

[krmaxwell]

OK I definitely see that the relationship is not being added, so something is up.

[markmaunu]

Let me know if I can provide anything else to debug . Are you uploading to 4-master ?

[webstergd]

Do you mind sending me a snippet of your Apache error log during this time?

I have seen this before when CRITs overloads. However the CRITs team was recently working on their parsers so I want to make sure that didn't change anything as well.

Usually I track with master. However, I am running some ML tests right now. So I am about a month behind what is current so I don't break anything. :)

Sent from my iPhone

On 05 Apr 2015, at 03:55, markmaunu notifications@github.com wrote:

Let me know if I can provide anything else to debug . Are you uploading to 4-master ?

— Reply to this email directly or view it on GitHub.

[markmaunu]

Sure : Below are the tail end of /var/log/apache2/error.log from running last night . My version of CRITs is definitely in sync with current 4_master.

[Sat Apr 04 17:03:37.642249 2015] [:error] [pid 16537:tid 139835754874624] Exception OSError: (2, 'No such file or directory', '/tmp/tmpsARj25.upload') in <bound method _TemporaryFileWrapper.del of <closed file '', mode 'w+b' at 0x7f2df37d8660>> ignored [Sat Apr 04 17:03:37.668567 2015] [:error] [pid 16537:tid 139835754874624] Exception OSError: (2, 'No such file or directory', '/tmp/tmpd0FPzq.upload') in <bound method _TemporaryFileWrapper.del of <closed file '', mode 'w+b' at 0x7f2df55dcae0>> ignored [Sat Apr 04 17:03:37.669050 2015] [:error] [pid 16537:tid 139835754874624] Exception OSError: (2, 'No such file or directory', '/tmp/tmpctSgRJ.upload') in <bound method _TemporaryFileWrapper.del of <closed file '', mode 'w+b' at 0x7f2df558d1e0>> ignored [Sat Apr 04 17:03:37.675781 2015] [:error] [pid 16537:tid 139835754874624] Exception OSError: (2, 'No such file or directory', '/tmp/tmpuz3Bcq.upload') in <bound method _TemporaryFileWrapper.del of <closed file '', mode 'w+b' at 0x7f2df55dcc00>> ignored [Sat Apr 04 17:03:37.686269 2015] [:error] [pid 16537:tid 139835754874624] Exception OSError: (2, 'No such file or directory', '/tmp/tmpipa4KR.upload') in <bound method _TemporaryFileWrapper.del of <closed file '', mode 'w+b' at 0x7f2df55dca50>> ignored [Sat Apr 04 18:58:29.999979 2015] [:error] [pid 8987:tid 139835704518400] [client 192.168.1.102:9502] mod_wsgi (pid=8987): Exception occurred processing WSGI script '/data/crits/django.wsgi'., referer: https://192.168.1.110/ [Sat Apr 04 18:58:30.000255 2015] [:error] [pid 8987:tid 139835704518400] [client 192.168.1.102:9502] IOError: failed to write data, referer: https://192.168.1.110/ [Sat Apr 04 18:58:35.527260 2015] [:error] [pid 8987:tid 139835830408960] [client 192.168.1.102:9504] mod_wsgi (pid=8987): Exception occurred processing WSGI script '/data/crits/django.wsgi'., referer: https://192.168.1.110/samples/details/ab90bae7260159d26869c79296b2cea2/ [Sat Apr 04 18:58:35.527321 2015] [:error] [pid 8987:tid 139835830408960] [client 192.168.1.102:9504] IOError: failed to write data, referer: https://192.168.1.110/samples/details/ab90bae7260159d26869c79296b2cea2/ [Sat Apr 04 18:59:57.175140 2015] [core:warn] [pid 8983:tid 139836067473280] AH00045: child process 8986 still did not exit, sending a SIGTERM [Sat Apr 04 18:59:59.178522 2015] [core:warn] [pid 8983:tid 139836067473280] AH00045: child process 8986 still did not exit, sending a SIGTERM [Sat Apr 04 19:00:01.181126 2015] [core:warn] [pid 8983:tid 139836067473280] AH00045: child process 8986 still did not exit, sending a SIGTERM [Sat Apr 04 19:00:03.184027 2015] [core:error] [pid 8983:tid 139836067473280] AH00046: child process 8986 still did not exit, sending a SIGKILL [Sat Apr 04 19:00:04.187743 2015] [mpm_event:notice] [pid 8983:tid 139836067473280] AH00491: caught SIGTERM, shutting down

[webstergd]

Something is going on. I dont like that IOError error you are receiving can you do: 1) tail -n 500 /var/log/apache2/error.log 2) tail -n 500 /logs/crits.log

btw you are welcome to send me the logs privately if you dont want to expose your system details. granted all your ip's are private so I doubt it matters all that much. Anyway, my email is webstergd@sec.in.tum.de

On Sun, Apr 5, 2015 at 5:38 PM, markmaunu notifications@github.com wrote:

Sure : Below are the tail end of /var/log/apache2/error.log from running last night . My version of CRITs is definitely in sync with current 4_master.

[Sat Apr 04 17:03:37.642249 2015] [:error] [pid 16537:tid 139835754874624] Exception OSError: (2, 'No such file or directory', '/tmp/tmpsARj25.upload') in ', mode 'w+b' at 0x7f2df37d8660>> ignored [Sat Apr 04 17:03:37.668567 2015] [:error] [pid 16537:tid 139835754874624] Exception OSError: (2, 'No such file or directory', '/tmp/tmpd0FPzq.upload') in ', mode 'w+b' at 0x7f2df55dcae0>> ignored [Sat Apr 04 17:03:37.669050 2015] [:error] [pid 16537:tid 139835754874624] Exception OSError: (2, 'No such file or directory', '/tmp/tmpctSgRJ.upload') in ', mode 'w+b' at 0x7f2df558d1e0>> ignored [Sat Apr 04 17:03:37.675781 2015] [:error] [pid 16537:tid 139835754874624] Exception OSError: (2, 'No such file or directory', '/tmp/tmpuz3Bcq.upload') in ', mode 'w+b' at 0x7f2df55dcc00>> ignored [Sat Apr 04 17:03:37.686269 2015] [:error] [pid 16537:tid 139835754874624] Exception OSError: (2, 'No such file or directory', '/tmp/tmpipa4KR.upload') in ', mode 'w+b' at 0x7f2df55dca50>> ignored [Sat Apr 04 18:58:29.999979 2015] [:error] [pid 8987:tid 139835704518400] [client 192.168.1.102:9502] mod_wsgi (pid=8987): Exception occurred processing WSGI script '/data/crits/django.wsgi'., referer: https://192.168.1.110/ [Sat Apr 04 18:58:30.000255 2015] [:error] [pid 8987:tid 139835704518400] [client 192.168.1.102:9502] IOError: failed to write data, referer: https://192.168.1.110/ [Sat Apr 04 18:58:35.527260 2015] [:error] [pid 8987:tid 139835830408960] [client 192.168.1.102:9504] mod_wsgi (pid=8987): Exception occurred processing WSGI script '/data/crits/django.wsgi'., referer: https://192.168.1.110/samples/details/ab90bae7260159d26869c79296b2cea2/ [Sat Apr 04 18:58:35.527321 2015] [:error] [pid 8987:tid 139835830408960] [client 192.168.1.102:9504] IOError: failed to write data, referer: https://192.168.1.110/samples/details/ab90bae7260159d26869c79296b2cea2/ [Sat Apr 04 18:59:57.175140 2015] [core:warn] [pid 8983:tid 139836067473280] AH00045: child process 8986 still did not exit, sending a SIGTERM [Sat Apr 04 18:59:59.178522 2015] [core:warn] [pid 8983:tid 139836067473280] AH00045: child process 8986 still did not exit, sending a SIGTERM [Sat Apr 04 19:00:01.181126 2015] [core:warn] [pid 8983:tid 139836067473280] AH00045: child process 8986 still did not exit, sending a SIGTERM [Sat Apr 04 19:00:03.184027 2015] [core:error] [pid 8983:tid 139836067473280] AH00046: child process 8986 still did not exit, sending a SIGKILL [Sat Apr 04 19:00:04.187743 2015] [mpm_event:notice] [pid 8983:tid 139836067473280] AH00491: caught SIGTERM, shutting down

— Reply to this email directly or view it on GitHub https://github.com/krmaxwell/maltrieve/issues/150#issuecomment-89794721.

[markmaunu]

Sure thing. I'll send them privately just not to muddy up the thread.

[webstergd]

Hey it looks like the issue is being caused by an overload on the CRITs server. mark and I are going to continue to monitor this and if this is really the case I will update the readme that discusses this.

Will keep you posted