search

csirtgadgets / massive-octo-spice

DEPRECATED - USE v3 (bearded-avenger)
https://github.com/csirtgadgets/bearded-avenger-deploymentkit/wiki
GNU Lesser General Public License v3.0
227 stars 62 forks source link

timeout when querying for ip address #409

Closed villain closed 8 years ago

villain commented 8 years ago

a couple of users have mentioned this on the IRC channel, i figured i would drop by notes here to see if this solves anyone elses problem. if you have a large number of indexes (eg; from when it was daily indexes), you might be experiencing issues querying for an IP address which look something like:

cif@cif:~$ cif -q 1.2.3.4 <!DOCTYPE html>

Server error

at /usr/local/bin/cif line 310.

the line number above seems to differ for people, but the error message is the same

you might also see something similar to the below in the /var/log/cif-starmon.log, although i'm not sure if its related yet: [Tue May 3 22:07:15 2016] [error] Can't use string ("0") as a HASH ref while "strict refs" in use at /opt/cif/bin/../lib/perl5/CIF/Client.pm line 225.

ive found a workaround to at least get data returned for queries for ip addresses is to throw --ttl=0 on the end, eg; cif -q 1.2.3.4 --ttl=0

what i'm unsure about is why using the cif command line to query would time out, however querying directly in elasticsearch seems to come back almost instantly;

GET cif.observables-*/observables/_search { "query" : { "term" : { "observable" : "1.2.3.4" } }

..

{ "took": 612, "timed_out": false, "_shards": { "total": 504, "successful": 504, "failed": 0 },

villain commented 8 years ago

probably should add; querying for other observables (domains, md5 etc) is fine

giovino commented 8 years ago

In 2.00.00-rc.13 there were several enhancements applied to address this issue. (https://github.com/csirtgadgets/massive-octo-spice/commit/1dafea3840beac59f1343ba4a7e732402e8aab89 https://github.com/csirtgadgets/massive-octo-spice/commit/1b0598827643306631f0dcf6bace7c0ca5996062 https://github.com/csirtgadgets/massive-octo-spice/commit/5f415069400010bc9f81a703f5c92aeebccf9c81)

Are you continuing to see this behavior at rc.13+

villain commented 8 years ago

i'm still on RC12, i'll close this and re-open if the problem still exists after the upgrade

priyamalik commented 7 years ago

I am also facing same error. I have installed CIF at two system (1 at ubuntu and 2nd on ubuntu-server). On 1st one every thing is fine but ubuntu server is throwing this error.

I have added --ttl=0 at the end but still same issue.