Problem with feed

[perez1987]

Hello,

I have a problem with feed. When CIF import this feed gives me this error:

cif-router.log:[2015-08-09T14:20:59,661Z][ERROR]: [Request] \ [http://172.20.106.91:80]-[599] Socket closed by remote server: Broken pipe, called from sub Search::Elastics earch::Transport::try {...} at /usr/share/perl5/Try/Tiny.pm line 81. With vars: {'status_code' => 599,'request' => {'ignore' => [],'serialize' => 'bulk','mime_type' => 'ap plication/json','method' => 'POST','body' =>

Why can it be? When running the test with CIF-smrt seems all right.

The yml conf this feed:

defaults: tlp: green altid: http://xxxx.com confidence: 60 altid_tlp: green tags:

• malware otype: ipv4 provider: xxxxxx pattern: ^(\S+)$ feeds: paps_mw: remote: /var/tmp/xxx values:
  • observable

Thank you very much

[giovino]

Q1: What is the version of the OS and what version[1] of CIF are you running?

Q2: What is the CPU, Ram, Disk specs?

Q3: Is this an all-in-one or did you by any chance split out ElasticSearch onto a separate box?

Q4: What do the Apache error logs report around the time of this error?

Q5: You say this works successfully with cif-smrt testmode? is this how you tested?

sudo su - cif -c "/opt/cif/bin/cif-smrt --testmode -c -d -r /etc/cif/rules/default/

[1] https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/bin/version.sh

[perez1987]

@giovino Problem solved! The file was very large and the machine is saturated

[giovino]

thanks for the follow up! Closing.

[perez1987]

@giovino The problem is back and not know how to fix it. The server resources are sufficient. Paste log error example

[2015-08-24T15:01:07,104Z][ERROR]: $VAR1 = '{ "portlist" : null, "reporttime" : "2015-08-24T13:01:04Z", "observable" : "mojonote.com", "tlp" : "green", "otype" : "fqdn", "altid_tlp" : "green", "provider" : "threatexchange.fb.com", "@version" : 2, "rdata" : [ "http://mojonote.com/note/public/227454/" ], "application" : null, "tags" : [ "malware" ], "firsttime" : "2015-08-24T13:01:05Z", "group" : [ "everyone" ], "protocol" : -1, "confidence" : 52.105, "@timestamp" : "2015-08-24T13:01:05.230Z", "id" : "7383ab4e069a6ce3b3c80cb42b7049cb1ef47c3c79c924cfdcee25db05947f12", "related" : "2247bea6a23fbd65026c667f9e60e94b307832687bffe666575ca4f88034bd82", "lang" : "EN", "altid" : "http://threatexchange.fb.com", "lasttime" : "2015-08-24T13:01:04Z" } '; [2015-08-24T15:01:07,862Z][ERROR]: zmq_send: Operation cannot be accomplished in current state at /usr/local/share/perl/5.18.2/ZMQ/FFI/ErrorHelper.pm line 49. ZMQ::FFI::ErrorHelper::fatal('ZMQ::FFI::ErrorHelper=HASH(0x266c478)', 'zmq_send') called at /usr/local/share/perl/5.18.2/ZMQ/FFI/ErrorHelper.pm line 28 ZMQ::FFI::ErrorHelper::check_error('ZMQ::FFI::ErrorHelper=HASH(0x266c478)', 'zmq_send', -1) called at (eval 1184) line 17 ZMQ::FFI::ErrorHandler::check_error('ZMQ::FFI::ZMQ3::Socket=HASH(0x5167720)', 'zmq_send', -1) called at /usr/local/share/perl/5.18.2/ZMQ/FFI/ZMQ3/Socket.pm line 30 ZMQ::FFI::ZMQ3::Socket::send('ZMQ::FFI::ZMQ3::Socket=HASH(0x5167720)', '{"version":2.0000001,"rtype":"submission","id":"443fa330ea5a5...') called at /opt/cif/bin/../lib/perl5/CIF/Worker.pm line 213 CIF::Worker::send('CIF::Worker=HASH(0x266c3d0)', 'ARRAY(0x53822e8)') called at /opt/cif/bin/../lib/perl5/CIF/Worker.pm line 173 CIF::Worker::process('CIF::Worker=HASH(0x266c3d0)', '{\x{a} "related" : "7a14f0d5906f543966431d75c83e8062605c097d709...') called at /opt/cif/bin/cif-worker line 269 main::try {...} () called at /usr/share/perl5/Try/Tiny.pm line 81 eval {...} called at /usr/share/perl5/Try/Tiny.pm line 72 Try::Tiny::try('CODE(0x53893b8)', 'Try::Tiny::Catch=REF(0x5495ef8)') called at /opt/cif/bin/cif-worker line 272 main::ANON('EV::IO=SCALAR(0x264e840)', 1) called at /usr/lib/perl5/AnyEvent/Impl/EV.pm line 88 eval {...} called at /usr/lib/perl5/AnyEvent/Impl/EV.pm line 88 AnyEvent::CondVar::Base::_wait('AnyEvent::CondVar=HASH(0x264e288)') called at /usr/lib/perl5/AnyEvent.pm line 1995 AnyEvent::CondVar::Base::recv('AnyEvent::CondVar=HASH(0x264e288)') called at /opt/cif/bin/cif-worker line 282 main::workers(4) called at /opt/cif/bin/cif-worker line 214 main::main() called at /opt/cif/bin/cif-worker line 183 main::ANON('Daemon::Control=HASH(0x50425c8)') called at /usr/local/share/perl/5.18.2/Daemon/Control.pm line 269 Daemon::Control::_launch_program('Daemon::Control=HASH(0x50425c8)') called at /usr/local/share/perl/5.18.2/Daemon/Control.pm line 223 Daemon::Control::_double_fork('Daemon::Control=HASH(0x50425c8)') called at /usr/local/share/perl/5.18.2/Daemon/Control.pm line 422 Daemon::Control::do_start('Daemon::Control=HASH(0x50425c8)') called at /usr/local/share/perl/5.18.2/Daemon/Control.pm line 616 Daemon::Control::run_command('Daemon::Control=HASH(0x50425c8)', 'start') called at /usr/local/share/perl/5.18.2/Daemon/Control.pm line 628 Daemon::Control::run('Daemon::Control=HASH(0x50425c8)') called at /opt/cif/bin/cif-worker line 185

Thank you

[giovino]

@perez1987

Thanks for the update, we tracking this in https://github.com/csirtgadgets/massive-octo-spice/issues/322