CIF server not providing out put file frequently.

[diveshshah]

Dear Team,

I have develop a CIF server with help of your guidelines.

Now we are facing frequent issue while running a query to get a output file. Our query is schedule on CIF server to provide output file, but its not providing output frequently.

When we restart our Server and run query again than Its provide output file. I am not able to understand what is concern behind an issue so i am sharing attach file of cif-smrt.log

Kindly provide solution and let me know if you require any other detail.

Thanks Divesh Shah

[wesyoung]

may want to start here:

https://github.com/csirtgadgets/massive-octo-spice/wiki/Troubleshooting-CIF

we run into this from time to time when the server is trying to return more results than it has memory for. usually it's an issue with elasticsearch not having enough memory, checking the elasticsearch and cif logs should highlight this.

[diveshshah]

Dear team,

I have gone through the link and follow the instruction or command need to run on CIF server. There is some command which is not giving similar output like document.

* 

tail /var/log/cif-router.log

* 

$ cif -q example.com -d

I am sharing some attachment for related to this, please go through and suggest, what need to do next for resolution.

1. 

tail /var/log/cif-router.log

2. 

tail /var/log/cif-router.lol

3. disk space 
4. elastic search 

Thanks Divesh Shah

From: "Wes" notifications@github.com To: "csirtgadgets" massive-octo-spice@noreply.github.com Cc: "diveshshah" divesh.shah@sequretek.com, "Author" author@noreply.github.com Sent: Friday, October 6, 2017 5:13:05 PM Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

may want to start here:

https://github.com/csirtgadgets/massive-octo-spice/wiki/Troubleshooting-CIF

we run into this from time to time when the server is trying to return more results than it has memory for. usually it's an issue with elasticsearch not having enough memory, checking the elasticsearch and cif logs should highlight this.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[diveshshah]

Dear team,

Any update on same.

Thanks Divesh Shah

From: "Divesh Shah" divesh.shah@sequretek.com To: "csirtgadgets" reply@reply.github.com Cc: "csirtgadgets" massive-octo-spice@noreply.github.com, "Author" author@noreply.github.com Sent: Tuesday, 10 October, 2017 16:33:02 Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

Dear team,

I have gone through the link and follow the instruction or command need to run on CIF server. There is some command which is not giving similar output like document.

* 

tail /var/log/cif-router.log

* 

$ cif -q example.com -d

I am sharing some attachment for related to this, please go through and suggest, what need to do next for resolution.

1. 

tail /var/log/cif-router.log

2. 

tail /var/log/cif-router.lol

3. disk space 
4. elastic search 

Thanks Divesh Shah

From: "Wes" notifications@github.com To: "csirtgadgets" massive-octo-spice@noreply.github.com Cc: "diveshshah" divesh.shah@sequretek.com, "Author" author@noreply.github.com Sent: Friday, October 6, 2017 5:13:05 PM Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

may want to start here:

https://github.com/csirtgadgets/massive-octo-spice/wiki/Troubleshooting-CIF

we run into this from time to time when the server is trying to return more results than it has memory for. usually it's an issue with elasticsearch not having enough memory, checking the elasticsearch and cif logs should highlight this.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[wesyoung]

i don't think github email captures attachments, so your screen shots aren't showing up. you may have to post them manually into the ticket.. what are your system specs (ram, cpu, etc).. like i said- it's prob too low for what you're trying todo..

[diveshshah]

Hi

Below are the required detail

As given two command of link is not giving out like same

tail /var/log/cif-router.log cif -q example.com -d

Some might helpfull for you to giving solution

tail /var/log/cif-router.log

[2017-10-24 17:56:32,528][INFO ][node ] [Orikal] initializing ... [2017-10-24 17:56:32,532][INFO ][plugins ] [Orikal] loaded [], sites [] [2017-10-24 17:56:35,637][INFO ][node ] [Orikal] initialized [2017-10-24 17:56:35,637][INFO ][node ] [Orikal] starting ... [2017-10-24 17:56:35,779][INFO ][transport ] [Orikal] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.2:9300]} [2017-10-24 17:56:35,796][INFO ][discovery ] [Orikal] elasticsearch/QW2uXQ1ETnqx8R58-vdzcQ [2017-10-24 17:56:39,561][INFO ][cluster.service ] [Orikal] new_master [Orikal][QW2uXQ1ETnqx8R58-vdzcQ][soc][inet[/192.168.0.2:9300]], reason: zen-disco-join (elected_as_master) [2017-10-24 17:56:39,610][INFO ][http ] [Orikal] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.0.2:9200]} [2017-10-24 17:56:39,610][INFO ][node ] [Orikal] started [2017-10-24 17:56:40,599][INFO ][gateway ] [Orikal] recovered [13] indices into cluster_state

free -m total used free shared buffers cached Mem: 7862 7304 557 112 89 1219 -/+ buffers/cache: 5994 1867 Swap: 3971 317 3654

df -h Filesystem Size Used Avail Use% Mounted on udev 3.9G 4.0K 3.9G 1% /dev tmpfs 787M 1.1M 786M 1% /run /dev/dm-0 455G 121G 311G 29% / none 4.0K 0 4.0K 0% /sys/fs/cgroup none 5.0M 0 5.0M 0% /run/lock none 3.9G 152K 3.9G 1% /run/shm none 100M 28K 100M 1% /run/user /dev/sda1 236M 74M 150M 33% /boot

tail /var/log/elasticsearch/elasticsearch.log

[2017-10-24 17:56:32,528][INFO ][node ] [Orikal] initializing ... [2017-10-24 17:56:32,532][INFO ][plugins ] [Orikal] loaded [], sites [] [2017-10-24 17:56:35,637][INFO ][node ] [Orikal] initialized [2017-10-24 17:56:35,637][INFO ][node ] [Orikal] starting ... [2017-10-24 17:56:35,779][INFO ][transport ] [Orikal] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.2:9300]} [2017-10-24 17:56:35,796][INFO ][discovery ] [Orikal] elasticsearch/QW2uXQ1ETnqx8R58-vdzcQ [2017-10-24 17:56:39,561][INFO ][cluster.service ] [Orikal] new_master [Orikal][QW2uXQ1ETnqx8R58-vdzcQ][soc][inet[/192.168.0.2:9300]], reason: zen-disco-join (elected_as_master) [2017-10-24 17:56:39,610][INFO ][http ] [Orikal] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.0.2:9200]} [2017-10-24 17:56:39,610][INFO ][node ] [Orikal] started [2017-10-24 17:56:40,599][INFO ][gateway ] [Orikal] recovered [13] indices into cluster_state

tail /var/log/cif-router.log

[2017-10-24T17:56:27,255Z][1715][FATAL]: [NoNodes] No nodes are available: [http://localhost:9200], called from sub Search::Elasticsearch::Transport::try {...} at /usr/share/perl5/Try/Tiny.pm line 81. [2017-10-24T17:56:27,255Z][1715][FATAL]: unable to start router, no storage handle... [2017-10-24T17:56:27,255Z][1715][WARN]: retrying in 5secs... [2017-10-24T17:56:32,257Z][1715][FATAL]: [NoNodes] No nodes are available: [http://localhost:9200], called from sub Search::Elasticsearch::Transport::try {...} at /usr/share/perl5/Try/Tiny.pm line 81. [2017-10-24T17:56:32,257Z][1715][FATAL]: unable to start router, no storage handle... [2017-10-24T17:56:32,257Z][1715][WARN]: retrying in 5secs... [2017-10-24T17:56:37,259Z][1715][FATAL]: [NoNodes] ** No nodes are available: [http://localhost:9200], called from sub Search::Elasticsearch::Transport::try {...} at /usr/share/perl5/Try/Tiny.pm line 81. [2017-10-24T17:56:37,259Z][1715][FATAL]: unable to start router, no storage handle... [2017-10-24T17:56:37,259Z][1715][WARN]: retrying in 5secs... [2017-10-24T17:56:42,673Z][1715][INFO]: started, waiting for messages..

curl -i 'http://localhost:9200/_cluster/health?pretty'

HTTP/1.1 200 OK Content-Type: application/json; charset=UTF-8 Content-Length: 280

{ "cluster_name" : "elasticsearch", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 65, "active_shards" : 65, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 65 }

Team check and provide solution of problem, let me know if you required any thing else.

Thanks Divesh Shah

From: "Wes" notifications@github.com To: "csirtgadgets" massive-octo-spice@noreply.github.com Cc: "diveshshah" divesh.shah@sequretek.com, "Author" author@noreply.github.com Sent: Monday, 16 October, 2017 18:04:31 Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

i don't think github email captures attachments, so your screen shots aren't showing up. you may have to post them manually into the ticket.. what are your system specs (ram, cpu, etc).. like i said- it's prob too low for what you're trying todo..

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[wesyoung]

it looks like your cif server isn't configured [for can't find?] your elasticsearch instance? (as per the cif-router logs suggesting they can't find your ES host..)

you probably need to set CIF_ES_HOST to your elasticsearch address in your ENV somewhere-

https://github.com/csirtgadgets/massive-octo-spice/blob/a6994e8725a06daaeea8f70578f0322f96346e72/hacking/packaging/ubuntu/init.d/cif-router#L53

so /etc/init.d/cif-router will pick it up and pass it to cif-router appropriately:

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/bin/cif-router#L28

[diveshshah]

Dear Team,

As i informed you our CIF server is giving required output from last 10th months.

From last two month we getting issue in query output frequently. Issue we faced is our CIF server is giving message sleeping :60min

Location: /var/log /cifsmart log

sleeping :60min

We have to restart our CIF server, to run query for getting proper output and it works. So what we should we do to resolve this issue.

Thanks Divesh Shah

From: "Wes" notifications@github.com To: "csirtgadgets" massive-octo-spice@noreply.github.com Cc: "diveshshah" divesh.shah@sequretek.com, "Author" author@noreply.github.com Sent: Wednesday, 25 October, 2017 18:23:25 Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

it looks like your cif server isn't configured [for can't find?] your elasticsearch instance? (as per the cif-router logs suggesting they can't find your ES host..)

you probably need to set CIF_ES_HOST to your elasticsearch address in your ENV somewhere-

https://github.com/csirtgadgets/massive-octo-spice/blob/a6994e8725a06daaeea8f70578f0322f96346e72/hacking/packaging/ubuntu/init.d/cif-router#L53

so /etc/init.d/cif-router will pick it up and pass it to cif-router appropriately:

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/bin/cif-router#L28

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[wesyoung]

cif-smrt only pulls feeds once an hour- which is why you see the sleeping message every 60min. you may want to check the index health of your elasticsearch cluster to make sure the last two month indices are green [and not yellow].

[diveshshah]

Thanks Will check and update you

Share exact path for that.

Also Is this issue is coming due to this message:

[2017-11-02T16:34:55,433Z][2125][INFO]: sleeping: 60min [2017-11-02T17:32:55,026Z][1920][INFO]: checking for router... [2017-11-02T17:34:55,501Z][1920][FATAL]: <!DOCTYPE html>

[2017-11-02T17:34:55,535Z][2125][INFO]: sleeping: 60min

From: "Wes" notifications@github.com To: "csirtgadgets" massive-octo-spice@noreply.github.com Cc: "diveshshah" divesh.shah@sequretek.com, "Author" author@noreply.github.com Sent: Thursday, November 2, 2017 5:51:30 PM Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

cif-smrt only pulls feeds once an hour- which is why you see the sleeping message every 60min. you may want to check the index health of your elasticsearch cluster to make sure the last two month indices are green [and not yellow].

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[wesyoung]

https://www.google.com/search?q=elasticsearch+cluster+health

[diveshshah]

Hi,

Below are the detail of health check of elastic search

soc@soc:~$ curl localhost:9200/_cat/health 1509689979 11:49:39 elasticsearch yellow 1 1 70 70 0 0 70

Please guide next step for resolution.

Thanks Divesh Shah

From: "Wes" notifications@github.com To: "csirtgadgets" massive-octo-spice@noreply.github.com Cc: "diveshshah" divesh.shah@sequretek.com, "Author" author@noreply.github.com Sent: Thursday, November 2, 2017 6:21:09 PM Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

https://www.google.com/search?q=elasticsearch+cluster+health

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[wesyoung]

https://www.google.com/search?q=how-to-fix-cluster-health-yellow-with-elasticsearch

[diveshshah]

Dear team,

So now i have to fix elasticsearch yellow into green. Will this resolve the concern of not getting feeds from CIF on some occasion.

Thanks Divesh Shah

From: "Wes" notifications@github.com To: "csirtgadgets" massive-octo-spice@noreply.github.com Cc: "diveshshah" divesh.shah@sequretek.com, "Author" author@noreply.github.com Sent: Friday, November 3, 2017 5:40:47 PM Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

https://www.google.com/search?q=how-to-fix-cluster-health-yellow-with-elasticsearch

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[diveshshah]

Dear team,

So now i have to fix elasticsearch yellow into green.

Please clear following points: 1.Will this resolve the concern of not getting feeds from CIF on some occasion.

2. Two Node creation required according to links, is any specific RAM dependency there for this.
3. If we resolve this issue of not getting feeds only this way, so can you please share exact required command to perform.

Thanks Divesh Shah

From: "Divesh Shah" divesh.shah@sequretek.com To: "csirtgadgets/massive-octo-spice" reply@reply.github.com Cc: "csirtgadgets" massive-octo-spice@noreply.github.com, "Author" author@noreply.github.com Sent: Wednesday, November 15, 2017 2:09:38 PM Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

Dear team,

So now i have to fix elasticsearch yellow into green. Will this resolve the concern of not getting feeds from CIF on some occasion.

Thanks Divesh Shah

From: "Wes" notifications@github.com To: "csirtgadgets" massive-octo-spice@noreply.github.com Cc: "diveshshah" divesh.shah@sequretek.com, "Author" author@noreply.github.com Sent: Friday, November 3, 2017 5:40:47 PM Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

https://www.google.com/search?q=how-to-fix-cluster-health-yellow-with-elasticsearch

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[wesyoung]

not sure- you'll need to test different configurations to see which works the best.

[diveshshah]

ok

Share me the exact link or step to create a node

Thanks Divesh Shah

From: "Wes" notifications@github.com To: "csirtgadgets" massive-octo-spice@noreply.github.com Cc: "diveshshah" divesh.shah@sequretek.com, "Author" author@noreply.github.com Sent: Wednesday, 15 November, 2017 18:38:02 Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

not sure- you'll need to test different configurations to see which works the best.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[diveshshah]

Dear team,

Waiting for Required step or exact steps need to perform to resolve this issue.

Thanks Divesh Shah

From: "Divesh Shah" divesh.shah@sequretek.com To: "csirtgadgets/massive-octo-spice" reply@reply.github.com Cc: "csirtgadgets" massive-octo-spice@noreply.github.com, "Author" author@noreply.github.com Sent: Wednesday, 22 November, 2017 10:41:31 Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

ok

Share me the exact link or step to create a node

Thanks Divesh Shah

From: "Wes" notifications@github.com To: "csirtgadgets" massive-octo-spice@noreply.github.com Cc: "diveshshah" divesh.shah@sequretek.com, "Author" author@noreply.github.com Sent: Wednesday, 15 November, 2017 18:38:02 Subject: Re: [csirtgadgets/massive-octo-spice] CIF server not providing out put file frequently. (#495)

not sure- you'll need to test different configurations to see which works the best.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[wesyoung]

this is out of scope for us, there is plenty of elasticsearch documentation out there that should help you address this. every environment is different so you may have to build out a test environment that meets your requirements and see what works.