search

csirtgadgets / massive-octo-spice

DEPRECATED - USE v3 (bearded-avenger)
https://github.com/csirtgadgets/bearded-avenger-deploymentkit/wiki
GNU Lesser General Public License v3.0
227 stars 62 forks source link

CIF Connection Issue #462

Closed priyamalik closed 7 years ago

priyamalik commented 7 years ago

My CIF was working fine, from from 2-3 days it was not showing output. So I ran command CIF -p and it is showing below error.

priyamalik commented 7 years ago

Also I ran below command to get output into csv file (Earlier it was giving output) but now it is showing below error. soc@soc:~$ cif --feed --otype ipv4 --today -f csv > testing.csv <!DOCTYPE html>

Server error

at /usr/local/bin/cif line 324.

wesyoung commented 7 years ago

anythin in the server logs?

i'm guessing you might be out-growing your elasticsearch instance...

fl0wc0ntr0l commented 7 years ago

Recently this has begun to happen to me when the machine re-boots or some cif-service (haven't nailed down which one) is restarted. I checked cif-tokens and both tokens I had set (one for the cif user and one for myself) are gone.

It's not the Elasticsearch instance size, either, because we're not keeping data older than 7 days.

wesyoung commented 7 years ago

what does the cluster status look like ?

fl0wc0ntr0l commented 7 years ago

Same as it always has.

curl -XGET 'http://localhost:9200/_cluster/health?pretty' { "cluster_name" : "elasticsearch", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 10, "active_shards" : 10, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 10 }

curl -XGET 'http://localhost:9200/_status?pretty' { "_shards" : { "total" : 20, "successful" : 10, "failed" : 0 }, "indices" : { "cif.observables-2016.11" : { "index" : { "primary_size_in_bytes" : 19160665, "size_in_bytes" : 19160665 }, "translog" : { "operations" : 346 }, "docs" : { "num_docs" : 9383, "max_doc" : 9383, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 84, "total_time_in_millis" : 13660, "total_docs" : 94318, "total_size_in_bytes" : 106343278 }, "refresh" : { "total" : 901, "total_time_in_millis" : 15391 }, "flush" : { "total" : 35, "total_time_in_millis" : 650 }, "shards" : { "0" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "SxpW6y1HST2icdMLZMJ3PA", "relocating_node" : null, "shard" : 0, "index" : "cif.observables-2016.11" }, "state" : "STARTED", "index" : { "size_in_bytes" : 3954298 }, "translog" : { "id" : 1480441704782, "operations" : 74 }, "docs" : { "num_docs" : 1927, "max_doc" : 1927, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 16, "total_time_in_millis" : 2773, "total_docs" : 18611, "total_size_in_bytes" : 21010009 }, "refresh" : { "total" : 174, "total_time_in_millis" : 3138 }, "flush" : { "total" : 7, "total_time_in_millis" : 128 } } ], "1" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "SxpW6y1HST2icdMLZMJ3PA", "relocating_node" : null, "shard" : 1, "index" : "cif.observables-2016.11" }, "state" : "STARTED", "index" : { "size_in_bytes" : 3737038 }, "translog" : { "id" : 1480441704854, "operations" : 63 }, "docs" : { "num_docs" : 1819, "max_doc" : 1819, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 17, "total_time_in_millis" : 2488, "total_docs" : 18472, "total_size_in_bytes" : 21138645 }, "refresh" : { "total" : 182, "total_time_in_millis" : 3514 }, "flush" : { "total" : 7, "total_time_in_millis" : 134 } } ], "2" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "SxpW6y1HST2icdMLZMJ3PA", "relocating_node" : null, "shard" : 2, "index" : "cif.observables-2016.11" }, "state" : "STARTED", "index" : { "size_in_bytes" : 3844885 }, "translog" : { "id" : 1480441704890, "operations" : 69 }, "docs" : { "num_docs" : 1862, "max_doc" : 1862, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 17, "total_time_in_millis" : 2690, "total_docs" : 18285, "total_size_in_bytes" : 20731590 }, "refresh" : { "total" : 181, "total_time_in_millis" : 3151 }, "flush" : { "total" : 7, "total_time_in_millis" : 95 } } ], "3" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "SxpW6y1HST2icdMLZMJ3PA", "relocating_node" : null, "shard" : 3, "index" : "cif.observables-2016.11" }, "state" : "STARTED", "index" : { "size_in_bytes" : 3924114 }, "translog" : { "id" : 1480441704819, "operations" : 81 }, "docs" : { "num_docs" : 1914, "max_doc" : 1914, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 17, "total_time_in_millis" : 3132, "total_docs" : 19388, "total_size_in_bytes" : 21625123 }, "refresh" : { "total" : 184, "total_time_in_millis" : 3419 }, "flush" : { "total" : 7, "total_time_in_millis" : 187 } } ], "4" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "SxpW6y1HST2icdMLZMJ3PA", "relocating_node" : null, "shard" : 4, "index" : "cif.observables-2016.11" }, "state" : "STARTED", "index" : { "size_in_bytes" : 3700330 }, "translog" : { "id" : 1480441704945, "operations" : 59 }, "docs" : { "num_docs" : 1861, "max_doc" : 1861, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 17, "total_time_in_millis" : 2577, "total_docs" : 19562, "total_size_in_bytes" : 21837911 }, "refresh" : { "total" : 180, "total_time_in_millis" : 2169 }, "flush" : { "total" : 7, "total_time_in_millis" : 106 } } ] } }, "cif.tokens" : { "index" : { "primary_size_in_bytes" : 8787, "size_in_bytes" : 8787 }, "translog" : { "operations" : 0 }, "docs" : { "num_docs" : 2, "max_doc" : 2, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 0, "total_time_in_millis" : 0, "total_docs" : 0, "total_size_in_bytes" : 0 }, "refresh" : { "total" : 7, "total_time_in_millis" : 119 }, "flush" : { "total" : 2, "total_time_in_millis" : 19 }, "shards" : { "0" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "SxpW6y1HST2icdMLZMJ3PA", "relocating_node" : null, "shard" : 0, "index" : "cif.tokens" }, "state" : "STARTED", "index" : { "size_in_bytes" : 4225 }, "translog" : { "id" : 1480441259928, "operations" : 0 }, "docs" : { "num_docs" : 1, "max_doc" : 1, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 0, "total_time_in_millis" : 0, "total_docs" : 0, "total_size_in_bytes" : 0 }, "refresh" : { "total" : 2, "total_time_in_millis" : 29 }, "flush" : { "total" : 1, "total_time_in_millis" : 10 } } ], "1" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "SxpW6y1HST2icdMLZMJ3PA", "relocating_node" : null, "shard" : 1, "index" : "cif.tokens" }, "state" : "STARTED", "index" : { "size_in_bytes" : 115 }, "translog" : { "id" : 1480441259931, "operations" : 0 }, "docs" : { "num_docs" : 0, "max_doc" : 0, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 0, "total_time_in_millis" : 0, "total_docs" : 0, "total_size_in_bytes" : 0 }, "refresh" : { "total" : 1, "total_time_in_millis" : 0 }, "flush" : { "total" : 0, "total_time_in_millis" : 0 } } ], "2" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "SxpW6y1HST2icdMLZMJ3PA", "relocating_node" : null, "shard" : 2, "index" : "cif.tokens" }, "state" : "STARTED", "index" : { "size_in_bytes" : 4217 }, "translog" : { "id" : 1480441259947, "operations" : 0 }, "docs" : { "num_docs" : 1, "max_doc" : 1, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 0, "total_time_in_millis" : 0, "total_docs" : 0, "total_size_in_bytes" : 0 }, "refresh" : { "total" : 2, "total_time_in_millis" : 90 }, "flush" : { "total" : 1, "total_time_in_millis" : 9 } } ], "3" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "SxpW6y1HST2icdMLZMJ3PA", "relocating_node" : null, "shard" : 3, "index" : "cif.tokens" }, "state" : "STARTED", "index" : { "size_in_bytes" : 115 }, "translog" : { "id" : 1480441259926, "operations" : 0 }, "docs" : { "num_docs" : 0, "max_doc" : 0, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 0, "total_time_in_millis" : 0, "total_docs" : 0, "total_size_in_bytes" : 0 }, "refresh" : { "total" : 1, "total_time_in_millis" : 0 }, "flush" : { "total" : 0, "total_time_in_millis" : 0 } } ], "4" : [ { "routing" : { "state" : "STARTED", "primary" : true, "node" : "SxpW6y1HST2icdMLZMJ3PA", "relocating_node" : null, "shard" : 4, "index" : "cif.tokens" }, "state" : "STARTED", "index" : { "size_in_bytes" : 115 }, "translog" : { "id" : 1480441259981, "operations" : 0 }, "docs" : { "num_docs" : 0, "max_doc" : 0, "deleted_docs" : 0 }, "merges" : { "current" : 0, "current_docs" : 0, "current_size_in_bytes" : 0, "total" : 0, "total_time_in_millis" : 0, "total_docs" : 0, "total_size_in_bytes" : 0 }, "refresh" : { "total" : 1, "total_time_in_millis" : 0 }, "flush" : { "total" : 0, "total_time_in_millis" : 0 } } ] } } } }

wesyoung commented 7 years ago

yellow is no good. means something is going on at the local level... depending on your setup (and amount of data) you may want to google around for "unassigned_shards" ... get an idea of what's going on. what i've seen in the past, when this happens tokens (for whatever reason) are the first thing to go... (something we're working to address more cleanly in v3 by way of doing more data aggregation firstseen/lastseen/count etc).

fl0wc0ntr0l commented 7 years ago

From the ES 1.4 KB on cluster health:

The cluster health status is: green, yellow or red. On the shard level, a red status indicates that the specific shard is not allocated in the cluster, yellow means that the primary shard is allocated but replicas are not, and green means that all shards are allocated.

We have not created replicas for this instance, which is probably why we are getting the yellow status, but I don't believe it's a problem right now as the data in CIF doesn't need to be particularly resilient.

wesyoung commented 7 years ago

dunno then. when i've seen tokens start 401'ing on me, usually elasticsearch is yellow and not returning the proper results. my hunch is you should try to get it into a green state and from there it'll be a little easier to figure out what the issue is... (i think).