search

csirtgadgets / massive-octo-spice

DEPRECATED - USE v3 (bearded-avenger)
https://github.com/csirtgadgets/bearded-avenger-deploymentkit/wiki
GNU Lesser General Public License v3.0
228 stars 62 forks source link

searches not being captured #257

Closed giovino closed 8 years ago

giovino commented 9 years ago

cif-beta-10

it does not appear searches are being captured:

user@ubuntu02:~$ cif -q example.com
user@ubuntu02:~$ cif -q example.com
user@ubuntu02:~$ cif -q example.com
user@ubuntu02:~$ cif -q example.com

Wes can you duplicate?

-g

jgedeon120 commented 9 years ago

In the current beta master we are also seeing this behavior quite a bit. Testing we have found that if adding a limit command to it we can then get results when without it thought there was none.

wesyoung commented 9 years ago

@jgedeon120 was this with the perl SDK client or the python SDK client?

jgedeon120 commented 9 years ago

Both.

On Mon, Jul 6, 2015 at 7:35 AM, Wes notifications@github.com wrote:

@jgedeon120 https://github.com/jgedeon120 was this with the perl SDK client or the python SDK client?

— Reply to this email directly or view it on GitHub https://github.com/csirtgadgets/massive-octo-spice/issues/257#issuecomment-118827441 .

Registered Linux User # 379282

jgedeon120 commented 9 years ago

Here is an interesting observation.

cif -vn -q 162.244.33.104 --limit 10 returns nothing.

cif -vn -q 162.244.33.104 returns plenty if you are bringing in osint.bambenekconsulting.com.

On Mon, Jul 6, 2015 at 7:54 AM, Joe Gedeon joe.gedeon@gmail.com wrote:

Both.

On Mon, Jul 6, 2015 at 7:35 AM, Wes notifications@github.com wrote:

@jgedeon120 https://github.com/jgedeon120 was this with the perl SDK client or the python SDK client?

— Reply to this email directly or view it on GitHub https://github.com/csirtgadgets/massive-octo-spice/issues/257#issuecomment-118827441 .

Registered Linux User # 379282

Registered Linux User # 379282

wesyoung commented 9 years ago

i think i know what's goin on, and i think it's limited to how we search specifically for ip addresses:

(cif)bender:cif-browsers wes$ cif -q secureserver.net
+-------+----------------------+------------------+-------+----+-----+----------+------------+-------------+-----------+-------+-----------+
|  tlp  |      reporttime      |    observable    | otype | cc | asn | asn_desc | confidence | description |    tags   | rdata | provider  |
+-------+----------------------+------------------+-------+----+-----+----------+------------+-------------+-----------+-------+-----------+
| green | 2015-06-23T12:06:12Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #234 | whitelist |       | alexa.com |
| green | 2015-06-23T23:12:32Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #233 | whitelist |       | alexa.com |
| green | 2015-06-24T00:12:12Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #233 | whitelist |       | alexa.com |
| green | 2015-06-25T00:12:09Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #233 | whitelist |       | alexa.com |
| green | 2015-06-25T02:12:45Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #235 | whitelist |       | alexa.com |
| green | 2015-06-26T00:12:11Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #235 | whitelist |       | alexa.com |
| green | 2015-06-26T03:12:38Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #232 | whitelist |       | alexa.com |
| green | 2015-06-27T00:12:15Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #232 | whitelist |       | alexa.com |
| green | 2015-06-27T02:12:15Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #229 | whitelist |       | alexa.com |
| green | 2015-06-30T10:52:21Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #226 | whitelist |       | alexa.com |
| green | 2015-06-30T23:25:18Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #225 | whitelist |       | alexa.com |
| green | 2015-07-01T00:25:16Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #225 | whitelist |       | alexa.com |
| green | 2015-07-02T00:08:51Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #225 | whitelist |       | alexa.com |
| green | 2015-07-02T10:20:16Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #223 | whitelist |       | alexa.com |
| green | 2015-07-03T00:20:35Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #223 | whitelist |       | alexa.com |
| green | 2015-07-03T09:20:15Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #219 | whitelist |       | alexa.com |
| green | 2015-07-03T23:20:35Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #220 | whitelist |       | alexa.com |
| green | 2015-07-04T00:20:36Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #220 | whitelist |       | alexa.com |
| green | 2015-07-04T23:20:18Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #221 | whitelist |       | alexa.com |
| green | 2015-07-05T00:20:18Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #221 | whitelist |       | alexa.com |
| green | 2015-07-05T21:04:15Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #225 | whitelist |       | alexa.com |
| green | 2015-07-06T00:04:13Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #225 | whitelist |       | alexa.com |
+-------+----------------------+------------------+-------+----+-----+----------+------------+-------------+-----------+-------+-----------+
(cif)bender:cif-browsers wes$ cif -q secureserver.net --limit 2
+-------+----------------------+------------------+-------+----+-----+----------+------------+-------------+-----------+-------+-----------+
|  tlp  |      reporttime      |    observable    | otype | cc | asn | asn_desc | confidence | description |    tags   | rdata | provider  |
+-------+----------------------+------------------+-------+----+-----+----------+------------+-------------+-----------+-------+-----------+
| green | 2015-07-05T21:04:15Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #225 | whitelist |       | alexa.com |
| green | 2015-07-06T00:04:13Z | secureserver.net |  fqdn |    |     |          |     50     |  alexa #225 | whitelist |       | alexa.com |
+-------+----------------------+------------------+-------+----+-----+----------+------------+-------------+-----------+-------+-----------+
giovino commented 8 years ago

I am not sure this is fixed:

bash massive-octo-spice/src/bin/version.sh -p
2.00.00-beta.12
user@ubuntu02:~$ cif -q example1.com
user@ubuntu02:~$ cif -q example1.com
user@ubuntu02:~$ cif -q example1.com
user@ubuntu02:~$ cif -q example1.com
user@ubuntu02:~$ cif -q example1.com
wesyoung commented 8 years ago 
vagrant@vagrant-ubuntu-trusty-64:/vagrant/p5-cif-sdk$ perl -Ilib bin/cif -q example.com
vagrant@vagrant-ubuntu-trusty-64:/vagrant/p5-cif-sdk$ perl -Ilib bin/cif -q example.com
tlp  |group   |reporttime          |observable |cc|asn|confidence|tags  |description|rdata|provider      |altid_tlp|altid
amber|everyone|2015-07-15T14:32:09Z|example.com|  |   |25        |search|           |     |root@localhost|         |     

vagrant@vagrant-ubuntu-trusty-64:/vagrant/p5-cif-sdk$ perl -Ilib bin/cif -q example.com
tlp  |group   |reporttime          |observable |cc|asn|confidence|tags  |description|rdata|provider      |altid_tlp|altid
amber|everyone|2015-07-15T14:32:09Z|example.com|  |   |25        |search|           |     |root@localhost|         |     
amber|everyone|2015-07-15T14:32:11Z|example.com|  |   |25        |search|           |     |root@localhost|         |

?

giovino commented 8 years ago

hmm.. I wonder if my CIF instance was in a funky state due to testing and subsequent (service restart|reboot) fixed the issue...

$ cif -q example3.com
tlp  |group   |reporttime          |observable  |cc|asn|confidence|tags  |description|rdata|provider      |altid_tlp|altid
amber|everyone|2015-07-15T14:54:07Z|example3.com|  |   |25        |search|           |     |root@localhost|         |     
amber|everyone|2015-07-15T14:54:09Z|example3.com|  |   |25        |search|           |     |root@localhost|         |     
amber|everyone|2015-07-15T14:54:11Z|example3.com|  |   |25        |search|           |     |root@localhost|         |     
amber|everyone|2015-07-15T14:54:13Z|example3.com|  |   |25        |search|           |     |root@localhost|         |     
amber|everyone|2015-07-15T14:54:15Z|example3.com|  |   |25        |search|           |     |root@localhost|         |     
amber|everyone|2015-07-15T14:54:18Z|example3.com|  |   |25        |search|           |     |root@localhost|         |     
amber|everyone|2015-07-15T14:54:23Z|example3.com|  |   |25        |search|           |     |root@localhost|         |     
amber|everyone|2015-07-15T14:54:37Z|example3.com|  |   |25        |search|           |     |root@localhost|         |

closing until reproducible.