search

csirtgadgets / bearded-avenger

CIF v3 -- the fastest way to consume threat intelligence
https://csirtgadgets.com/collective-intelligence-framework
Mozilla Public License 2.0
183 stars 51 forks source link

cif client failure to return data #385

Closed sven342 closed 6 years ago

sven342 commented 6 years ago

cif v3.0.0b5 Potentially related to #383. The hotfix works for ASN. Several other combinations do not return data with an ES backend. These may all have the same cause but a few combinations I have tested include:

"cif --tags phishing" returns results but when combined with an itype: "cif --tags phishing --itype ipv4" no results are returned.

"cif --provider phishtank.com --itype ipv4 --tags phishing" returns no results "cif --last-day" returns no results "cif --today" returns no results "cif --feed --itype ipv4 --tags phishing" returns no results

wesyoung commented 6 years ago

"cif --tags phishing" returns results but when combined with an itype: "cif --tags phishing --itype ipv4" no results are returned.

try this with --confidence 5 and lmk if you see something

"cif --provider phishtank.com --itype ipv4 --tags phishing" returns no results

try this with --confidence 5 and lmk if you see something

"cif --last-day" returns no results "cif --today" returns no results

paste the results of -d with that, also test with some --tags

"cif --feed --itype ipv4 --tags phishing" returns no results

same thing, --confidence 5

sven342 commented 6 years ago 
cif --tags phishing --itype ipv4 --confidence 5
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
| tlp | group | reporttime | indicator | firsttime | lasttime | count | tags | description | confidence | rdata | provider |
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
 cif --provider phishtank.com --itype ipv4 --tags phishing --confidence 5
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
| tlp | group | reporttime | indicator | firsttime | lasttime | count | tags | description | confidence | rdata | provider |
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
cif --last-day -d
2018-03-20 19:42:39,491 - DEBUG - urllib3.connectionpool[208][MainThread] - Starting new HTTP connection (1): localhost
2018-03-20 19:42:39,521 - DEBUG - urllib3.connectionpool[396][MainThread] - http://localhost:5000 "GET /search?nolog=False&days=1&limit=500 HTTP/1.1" 200 39
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
| tlp | group | reporttime | indicator | firsttime | lasttime | count | tags | description | confidence | rdata | provider |
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
cif --today -d
2018-03-20 19:46:22,965 - DEBUG - urllib3.connectionpool[208][MainThread] - Starting new HTTP connection (1): localhost
2018-03-20 19:46:23,004 - DEBUG - urllib3.connectionpool[396][MainThread] - http://localhost:5000 "GET /search?nolog=False&reporttime=2018-03-20T00%3A00%3A00Z&limit=500 HTTP/1.1" 200 39
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
| tlp | group | reporttime | indicator | firsttime | lasttime | count | tags | description | confidence | rdata | provider |
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
cif --today --tags phishing -d
2018-03-20 19:47:03,440 - DEBUG - urllib3.connectionpool[208][MainThread] - Starting new HTTP connection (1): localhost
2018-03-20 19:47:03,466 - DEBUG - urllib3.connectionpool[396][MainThread] - http://localhost:5000 "GET /search?tags=phishing&nolog=False&reporttime=2018-03-20T00%3A00%3A00Z&limit=500 HTTP/1.1" 200 39
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
| tlp | group | reporttime | indicator | firsttime | lasttime | count | tags | description | confidence | rdata | provider |
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
cif --feed --itype ipv4 --tags phishing --confidence 5
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
| tlp | group | reporttime | indicator | firsttime | lasttime | count | tags | description | confidence | rdata | provider |
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
+-----+-------+------------+-----------+-----------+----------+-------+------+-------------+------------+-------+----------+
wesyoung commented 6 years ago

did you try walking down the confidence to 0? a lot of times ipv4 stuff from resolved urls will be in the 0-4 range. also check the logs, if not, fill out the issue template and look at:

https://github.com/csirtgadgets/bearded-avenger-deploymentkit/wiki/FAQ#searching-logs

sven342 commented 6 years ago

Yes I have tried walking the confidence down. I have csirtg-smrt and cif-router in debug mode but no log entries are generated referencing the cif client queries.

wesyoung commented 6 years ago

do you have hunters turned on?

https://github.com/csirtgadgets/bearded-avenger-deploymentkit/wiki/FAQ#what-is-a-hunter

that may explain the lack of lower level ips for things like phishing urls..

sven342 commented 6 years ago

I do... I was just using that as an example. So if I run "cif" without any arguments I get results, but any of those above do not return results. I would think "cif --today" would likely return the same results as "cif" but it does not return any results. This is a week old easy button install with the only change being ES is used as the datastore.

wesyoung commented 6 years ago

and when you see data

(do you have openly sharable samples?)

sven342 commented 6 years ago

Yes to both. Even stranger, cif --today has started to return a few results (but only of confidence level of 9?) and does not seem to match just a plain cif listing.

cif --today --limit 10
+-------+----------+----------------------------+-----------------+----------------------------+----------------------------+-------+--------------------+----------------------------------+------------+-------+---------------+
|  tlp  |  group   |         reporttime         |    indicator    |         firsttime          |          lasttime          | count |        tags        |           description            | confidence | rdata | provider      |
+-------+----------+----------------------------+-----------------+----------------------------+----------------------------+-------+--------------------+----------------------------------+------------+-------+---------------+
| green | everyone | 2018-03-27T00:00:00.11110Z |  185.39.216.15  | 2018-03-26T22:27:11.00000Z | 2018-03-27T00:00:16.93117Z |   1   |      exploit       | cbl + customised njabl. 3rd pa.. |    9.0     |       | spamhaus.org  |
| green | everyone | 2018-03-27T00:00:00.11253Z |   46.8.158.82   | 2018-03-26T22:57:14.00000Z | 2018-03-26T22:57:14.00000Z |   1   | bruteforce,scanner |                                  |    9.0     |       | dataplane.org |
| green | everyone | 2018-03-27T00:00:00.11979Z |  185.189.58.160 | 2018-03-26T22:05:26.00000Z | 2018-03-26T22:05:26.00000Z |   1   | bruteforce,scanner |                                  |    9.0     |       | dataplane.org |
| green | everyone | 2018-03-27T00:00:00.11979Z |  185.189.58.160 | 2018-03-26T22:05:26.00000Z | 2018-03-27T00:00:16.93888Z |   1   |      exploit       | cbl + customised njabl. 3rd pa.. |    9.0     |       | spamhaus.org  |
| green | everyone | 2018-03-27T00:00:00.12156Z | 185.190.103.162 | 2018-03-26T22:59:58.00000Z | 2018-03-26T22:59:58.00000Z |   1   | bruteforce,scanner |                                  |    9.0     |       | dataplane.org |
| green | everyone | 2018-03-27T00:00:00.12156Z | 185.190.103.162 | 2018-03-26T22:59:58.00000Z | 2018-03-27T00:00:17.24033Z |   1   |      exploit       | cbl + customised njabl. 3rd pa.. |    9.0     |       | spamhaus.org  |
| green | everyone | 2018-03-27T00:00:00.12909Z |  131.72.216.146 | 2018-03-26T22:36:40.00000Z | 2018-03-26T22:36:40.00000Z |   1   | bruteforce,scanner |                                  |    9.0     |       | dataplane.org |
| green | everyone | 2018-03-27T00:00:00.12909Z |  131.72.216.146 | 2018-03-26T22:36:40.00000Z | 2018-03-27T00:00:17.07485Z |   1   |      exploit       | cbl + customised njabl. 3rd pa.. |    9.0     |       | spamhaus.org  |
| green | everyone | 2018-03-27T00:00:00.19463Z |  66.118.165.127 | 2018-03-26T22:27:58.00000Z | 2018-03-26T22:27:58.00000Z |   1   | bruteforce,scanner |                                  |    9.0     |       | dataplane.org |
| green | everyone | 2018-03-27T00:00:00.19463Z |  66.118.165.127 | 2018-03-26T22:27:58.00000Z | 2018-03-27T00:00:17.46496Z |   1   |      exploit       | cbl + customised njabl. 3rd pa.. |    9.0     |       | spamhaus.org  |
+-------+----------+----------------------------+-----------------+----------------------------+----------------------------+-------+--------------------+----------------------------------+------------+-------+---------------+
cif --limit 10
+-------+----------+----------------------------+-----------------+----------------------------+----------------------------+-------+-----------------------------+----------------------------------+------------+-------+----------------+
|  tlp  |  group   |         reporttime         |    indicator    |         firsttime          |          lasttime          | count |             tags            |           description            | confidence | rdata | provider       |
+-------+----------+----------------------------+-----------------+----------------------------+----------------------------+-------+-----------------------------+----------------------------------+------------+-------+----------------+
| green | everyone | 2018-03-27T18:00:55.71436Z |   80.82.77.33   | 2018-03-27T12:53:01.00000Z | 2018-03-27T12:53:01.00000Z |   1   | honeynet,scanner,suspicious |         honeypot traffic         |    8.0     |       | packetmail.net |
| green | everyone | 2018-03-27T18:00:55.71472Z |  27.15.152.221  | 2018-03-27T12:53:12.00000Z | 2018-03-27T12:53:12.00000Z |   1   | honeynet,scanner,suspicious |         honeypot traffic         |    8.0     |       | packetmail.net |
| green | everyone | 2018-03-27T18:00:55.71436Z |   80.82.77.33   | 2018-03-27T12:53:01.00000Z | 2018-03-27T18:01:05.52417Z |   1   |           exploit           | cbl + customised njabl. 3rd pa.. |    9.0     |       | spamhaus.org   |
| green | everyone | 2018-03-27T18:00:55.71507Z |  107.170.200.10 | 2018-03-27T12:53:56.00000Z | 2018-03-27T12:53:56.00000Z |   1   | honeynet,scanner,suspicious |         honeypot traffic         |    8.0     |       | packetmail.net |
| green | everyone | 2018-03-27T18:00:55.71546Z |  77.241.49.154  | 2018-03-27T12:54:36.00000Z | 2018-03-27T12:54:36.00000Z |   1   | honeynet,scanner,suspicious |         honeypot traffic         |    8.0     |       | packetmail.net |
| green | everyone | 2018-03-27T18:00:55.71587Z | 163.172.168.251 | 2018-03-27T12:54:50.00000Z | 2018-03-27T12:54:50.00000Z |   1   | honeynet,scanner,suspicious |         honeypot traffic         |    8.0     |       | packetmail.net |
| green | everyone | 2018-03-27T18:00:55.71507Z |  107.170.200.10 | 2018-03-27T12:53:56.00000Z | 2018-03-27T18:01:05.61325Z |   1   |           exploit           | cbl + customised njabl. 3rd pa.. |    9.0     |       | spamhaus.org   |
| green | everyone | 2018-03-27T18:00:55.71629Z | 177.189.191.208 | 2018-03-27T12:54:59.00000Z | 2018-03-27T12:54:59.00000Z |   1   | honeynet,scanner,suspicious |         honeypot traffic         |    8.0     |       | packetmail.net |
| green | everyone | 2018-03-27T18:00:58.91269Z |  131.161.77.252 | 2018-03-27T12:13:46.00000Z | 2018-03-27T12:13:46.00000Z |   1   | honeynet,scanner,suspicious |         honeypot traffic         |    8.0     |       | packetmail.net |
| green | everyone | 2018-03-27T18:00:58.91269Z |  131.161.77.252 | 2018-03-27T12:13:46.00000Z | 2018-03-27T18:01:05.83401Z |   1   |           exploit           | cbl + customised njabl. 3rd pa.. |    9.0     |       | spamhaus.org   |
+-------+----------+----------------------------+-----------------+----------------------------+----------------------------+-------+-----------------------------+----------------------------------+------------+-------+----------------+
wesyoung commented 6 years ago

what do you get if you do this:

$ cif --itype ipv4 --confidence 0 --no-feed --tags phishing --limit 10

and can you verify that CIF_HUNTER_THREADS=2 AND CIF_HUNTER_ADVANCED=1 are in /etc/cif.env (and cif-router was restarted) ?

what you're seeing in the last two commands cif --today --limit.. is normal, the lack of --no-feed is applying a default confidence limit of 8, and it looks like data is coming in as you queried. what you're seeing is scanner data (highly confident) come in from other feeds, but phishing ipv4 data (which carries a much lower confidence) doesn't' look like it's being created (eg: hunters aren't doing what they're supposed to, maybe).

does that make sense?

sven342 commented 6 years ago 
+-------+----------+----------------------------+----------------+----------------------------+----------------------------+-------+----------+--------------------+------------+----------------------------------+---------------+
|  tlp  |  group   |         reporttime         |   indicator    |         firsttime          |          lasttime          | count |   tags   |    description     | confidence |              rdata               | provider      |
+-------+----------+----------------------------+----------------+----------------------------+----------------------------+-------+----------+--------------------+------------+----------------------------------+---------------+
| green | everyone | 2018-03-29T15:01:49.32528Z | 75.127.10.159  | 2018-03-29T11:48:06.00000Z | 2018-03-29T15:02:21.73261Z |   1   | phishing | abused legit phish |    4.0     | secured-acvity.paypa1.com.verf.. | spamhaus.org  |
| green | everyone | 2018-03-29T15:01:49.32528Z | 75.127.10.159  | 2018-03-29T11:48:06.00000Z | 2018-03-29T15:02:21.73590Z |   1   | phishing | abused legit phish |    2.0     | secured-acvity.paypa1.com.verf.. | spamhaus.org  |
| green | everyone | 2018-03-29T15:01:49.32625Z | 162.210.99.162 | 2018-03-29T08:47:29.00000Z | 2018-03-29T15:02:03.03081Z |   1   | phishing |       other        |    2.0     |  dropbox.digitalsurveyorske.com  | phishtank.com |
| green | everyone | 2018-03-29T15:01:49.33732Z |  104.28.27.56  | 2018-03-29T07:33:55.00000Z | 2018-03-29T15:02:04.12026Z |   1   | phishing |       other        |    2.0     |        kampanyagiris.com         | phishtank.com |
| green | everyone | 2018-03-29T15:01:49.33732Z |  104.28.26.56  | 2018-03-29T07:33:55.00000Z | 2018-03-29T15:02:04.12116Z |   1   | phishing |       other        |    2.0     |        kampanyagiris.com         | phishtank.com |
| green | everyone | 2018-03-29T15:01:49.36662Z | 145.14.145.161 | 2018-03-28T19:01:01.00000Z | 2018-03-29T15:02:03.32403Z |   1   | phishing |       other        |    2.0     | servicegroupesmsetmms.comli.co.. | phishtank.com |
| green | everyone | 2018-03-29T15:01:49.38988Z | 198.71.232.10  | 2018-03-28T13:44:57.00000Z | 2018-03-29T15:02:03.69521Z |   1   | phishing |       other        |    2.0     |    orange13.godaddysites.com     | phishtank.com |
| green | everyone | 2018-03-29T15:01:49.43527Z |  96.47.40.25   | 2018-03-28T06:44:29.00000Z | 2018-03-29T15:02:06.77811Z |   1   | phishing |       other        |    2.0     |      waterconflictforum.org      | phishtank.com |
| green | everyone | 2018-03-29T15:01:49.81882Z | 104.24.103.168 | 2018-03-25T17:12:02.00000Z | 2018-03-29T15:02:04.83944Z |   1   | phishing |       paypal       |    2.0     | www.pp-kontaktsystem-meldungal.. | phishtank.com |
| green | everyone | 2018-03-29T15:01:49.81882Z | 104.24.102.168 | 2018-03-25T17:12:02.00000Z | 2018-03-29T15:02:04.84044Z |   1   | phishing |       paypal       |    2.0     | www.pp-kontaktsystem-meldungal.. | phishtank.com |
+-------+----------+----------------------------+----------------+----------------------------+----------------------------+-------+----------+--------------------+------------+----------------------------------+---------------+

Ok, I think I understand now what's going on. I do have the cif hunter threads and advanced enabled in cif.env and restarted cif-router.

wesyoung commented 6 years ago

yea hunters [and the data they generate intersected with feeds] are a bit less documented because they tend to trip people up who aren't used to that dimension of data (eg: how do we treat confidence of something when we resolved a url to an ip address and we want to make sure that IP doesn't get into a feed by accident). additionally that the cif client does some magic by default- because we have users who just "gimme data i throw into firewall WTF U BLOCK NETFLIX", cause even if we wrote doc, who reads doc .. until you block netflix. chicken and egg problem.

i'm actually working on a blog piece for this week in that direction (confidence, etc) which leads up into better hunter doc in v4.