cif query Malformed request

[coonsmatthew]

Hello,

I've recently noticed that one of my CIF feeds seems to have broken and I'm trying to figure out why.

I'm currently generating three feeds for BRO, using output types of ipv4, fqdn, and url. The fqdn and url searches work, but the ipv4 search returns an error that says Malformed request at /usr/local/bin/cif line 321.

Here are the three searches (exactly the same minus the output type)

cif --token 12345 --otype url --feed --confidence 85 --format bro --tags phishing,botnet,exploit,feodo,gozi,hijacked,malware,rdata,scanner,search,zeus > /datadirectory/results.intel

cif --token 12345 --otype fqdn --feed --confidence 85 --format bro --tags phishing,botnet,exploit,feodo,gozi,hijacked,malware,rdata,scanner,search,zeus > /datadirectory/results.intel

cif --token 12345 --otype ipv4 --feed --confidence 85 --format bro --tags phishing,botnet,exploit,feodo,gozi,hijacked,malware,rdata,scanner,search,zeus > /datadirectory/results.intel

When I run the search manually with --debug enabled, here's what I see:

[2016-09-23T09:53:56,235Z][INFO][main:268]: starting up client... [2016-09-23T09:53:56,236Z][INFO][main:303]: running search... [2016-09-23T09:53:56,237Z][DEBUG][CIF::SDK::Client:170]: uri created: https://localhost/observables?gzip=1&tags=phishing,botnet,exploit,feodo,gozi,hijacked,malware,rdata,scanner,search,zeus&reporttimeend=2016-09-23T13:53:56Z&reporttime=2016-08-24T13:53:56Z&confidence=85&limit=50000&otype=ipv4 [2016-09-23T09:53:56,237Z][DEBUG][CIF::SDK::Client:171]: making request... [2016-09-23T09:54:26,779Z][INFO][CIF::SDK::Client:175]: status: 503 [2016-09-23T09:54:26,779Z][INFO][CIF::SDK::Client:181]: response size: < 1MB [2016-09-23T09:54:26,780Z][DEBUG][CIF::SDK::Client:184]: decoding content..

[2016-09-23T09:54:26,780Z][DEBUG][CIF::SDK::Client:193]: Data input to gunzip is not in gzip format at /usr/local/share/perl/5.18.2/CIF/SDK/Client.pm line 189. Malformed request at /usr/local/bin/cif line 321.

What am I doing wrong with this search?

[coonsmatthew]

An update to this...it appears that the the issue presents itself when I include the --feed option but only for --otype ipv4 queries.

Additionally, I get this error when running this search:

cif --otype ipv4 --confidence=85 --limit=5 --feed

Debug below:

$ cif --otype ipv4 --confidence=85 --limit=5 --feed --debug [2016-09-26T11:07:03,429Z][INFO][main:268]: starting up client... [2016-09-26T11:07:03,429Z][INFO][main:303]: running search... [2016-09-26T11:07:03,431Z][DEBUG][CIF::SDK::Client:170]: uri created: https://localhost/observables?otype=ipv4&gzip=1&reporttimeend=2016-09-26T15:07:03Z&limit=5&confidence=85&reporttime=2016-08-27T15:07:03Z [2016-09-26T11:07:03,431Z][DEBUG][CIF::SDK::Client:171]: making request... [2016-09-26T11:07:08,432Z][INFO][CIF::SDK::Client:175]: status: 200 [2016-09-26T11:07:08,433Z][INFO][CIF::SDK::Client:181]: response size: < 1MB [2016-09-26T11:07:08,433Z][DEBUG][CIF::SDK::Client:184]: decoding content..

[2016-09-26T11:07:08,433Z][DEBUG][main:334]: pulling ipv4 whitelist [2016-09-26T11:07:08,433Z][DEBUG][CIF::SDK::Client:170]: uri created: https://localhost/observables?confidence=25&limit=50000&otype=ipv4&gzip=1&reporttimeend=2016-09-26T15:07:03Z&reporttime=2016-08-27T15:07:03Z&tags=whitelist [2016-09-26T11:07:08,434Z][DEBUG][CIF::SDK::Client:171]: making request... [2016-09-26T11:07:32,660Z][INFO][CIF::SDK::Client:175]: status: 503 [2016-09-26T11:07:32,661Z][INFO][CIF::SDK::Client:181]: response size: < 1MB [2016-09-26T11:07:32,661Z][DEBUG][CIF::SDK::Client:184]: decoding content..

[2016-09-26T11:07:32,662Z][DEBUG][CIF::SDK::Client:193]: Data input to gunzip is not in gzip format at /usr/local/share/perl/5.18.2/CIF/SDK/Client.pm line 189. [2016-09-26T11:07:32,662Z][ERROR][main:346]: Malformed request

[wesyoung]

i think this might be an issue with the perl client specifically, are you able to test with the python client (on a sep vm somewhere) to verify it's a client issue and not a local data issue?

[coonsmatthew]

So...I started noticing this issue on queries that I had no issues with before. So I rebooted the server, and now everything's working great.

I will keep an eye on it to see if the issue presents itself again. If it does, I will try the Python client. Right now we're just generating threat feeds directly on the CIF server, and thus using the perl client to generate the feeds.

[wesyoung]

aye; wonder if elasticsearch was getting cranky...

https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html

if it happens again; try that and see if it's yellow or red. sometimes the errors can be a little mis-leading (trying to clean that up in v3).

[coonsmatthew]

So, interestingly, I'm seeing additional logs in the CIF-Router.log about this error. It re-appeared this morning.

Here is the log line that corresponds to the query I attempted:

[2016-10-04T09:11:37,679Z][26875][ERROR]: [Request] \ [http://localhost:9200]-[500] {"error":"OutOfMemoryError[Java heap space]","status":500}, called from sub Search::Elasticsearch::Transport::try {...} at /usr/share/perl5/Try/Tiny.pm line 81. With vars: {'status_code' => 500,'request' => {'serialize' => 'std','mime_type' => 'application/json','path' => '/cif.observables-%2A/_search','method' => 'GET','body' => {'sort' => [{'reporttime' => {'order' => 'desc'}}],'query' => {'filtered' => {'filter' => {'and' => [{'or' => [{'term' => {'tags' => ['whitelist']}}]},{'or' => [{'term' => {'otype' => ['ipv4']}}]},{'or' => [{'term' => {'group' => ['group.group']}},{'term' => {'group' => ['GROUP']}},{'term' => {'group' => ['GROUP']}},{'term' => {'group' => ['GROUP']}}]},{'range' => {'confidence' => {'gte' => '25'}}},{'range' => {'reporttime' => {'gte' => '2016-09-04T13:11:01Z','lte' => '2016-10-04T13:11:01Z'}}}]},'query' => {'match_all' => {}}}}},'ignore' => [],'qs' => {'timeout' => '30000000','size' => 50000}}}

I'm only using 5gb of ram out of 16gb available and I'm not even using any swap.

Interestingly I see no corresponding error in the elasticsearch.log file.

My elasticsearch cluster is yellow, i'm guessing because I've only got one node, I'm not running in a distributed model right now.

{ "cluster_name" : "elasticsearch", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 396, "active_shards" : 396, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 396 }

[wesyoung]

it just means you probably need to tune your ES instance. we had started modeling some of this as shown by these ansible templates:

https://github.com/csirtgadgets/massive-octo-spice/tree/develop/ansible/roles/debops.elasticsearch/templates/etc

https://www.elastic.co/guide/en/elasticsearch/reference/1.4/setup-configuration.html

specifically what you're probably trying to figure out is:

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/ansible/roles/debops.elasticsearch/templates/etc/default/elasticsearch.j2

(ie: giving java more room to roam)

[coonsmatthew]

Thanks Wes! I've put these suggestions into play on our server.