Parser for: Multiple observables in CSV file

[epazoglu]

Hello i have multiple observables included file,

as seen below.

FR,37.187.154.90,55ebc19ba7db43c08eb0ee4e7a5f7af58fd81159c39e38210fc04b7871677a0a,http://forexinsuracembard.com/j97S0E.php,11/4/2015 12:54:00 PM
FR,37.187.154.90,acb4b7a5647b5c7b57fa8f45cf2ebb874e1e90b036eceb3337409c200941b120,http://forexinsuracembard.com/j97S0E.php,11/4/2015 12:02:00 AM
FR,51.254.207.181,480a80823cf07e3b44fdf472932e1cc7b124f97a90b5b4255ac51e2c9c33adf0,http://zemamranews.com/jxke9u.php,11/4/2015 3:32:00 AM
FR,51.254.207.181,691c91b35ef3a73779d7f087d4992dd5f0794ebc38841acfe2767aa8cd6ae9ec,http://zemamranews.com/jxke9u.php,11/4/2015 7:36:00 AM
FR,91.216.107.152,55ebc19ba7db43c08eb0ee4e7a5f7af58fd81159c39e38210fc04b7871677a0a,http://abenorbenin.com/jcMISv.php,11/4/2015 12:54:00 PM
FR,91.216.107.152,c35e8ff5c19ee953ccb86f2c9f649160b6c7e5649318f3e5c081c90f266c6021,http://abenorbenin.com/jcMISv.php,11/4/2015 1:09:00 PM
FR,193.37.145.25,48cd9110979c03e7a121a35d8d452f971e7f3c93cc751bce12700eedfdbda27d,http://tmp3malinium.com/7DSCmu.php,11/4/2015 9:44:00 PM
FR,193.37.145.25,2101478b684bc6127bd03443d509374f8a53a254dcc5bdb076219ca98007b8b4,http://tmp3malinium.com/7DSCmu.php,11/4/2015 2:56:00 PM
FR,193.37.145.77,55ebc19ba7db43c08eb0ee4e7a5f7af58fd81159c39e38210fc04b7871677a0a,http://gainsenligne.info/TiWyMt.php,11/4/2015 12:54:00 PM
FR,193.37.145.77,acb4b7a5647b5c7b57fa8f45cf2ebb874e1e90b036eceb3337409c200941b120,http://gainsenligne.info/TiWyMt.php,11/4/2015 12:02:00 AM
DE,46.101.235.249,ab0123cb64f3314376c6da2fa5b6639ebe87580cbf427ab7fd66563c75148ddd,http://naimselmonaj.com/QoYx31.php,11/4/2015 12:40:00 AM
DE,79.140.41.112,c35e8ff5c19ee953ccb86f2c9f649160b6c7e5649318f3e5c081c90f266c6021,http://ipmon.net/CLuOIk.php,11/4/2015 1:09:00 PM

i write parser file

parser: csv
defaults:
  provider: http://cyberthreatalliance.org
  altid_tlp: amber
  tlp: amber
  confidence: 75
  values:
    - cc
    - observable
    - observable
    - observable
    - lasttime

feeds:
  CTA_cryptowall:
    remote: /opt/Cyber_Threat_Alliance_Cryptowall-2.csv
    tags:
      - botnet
      - malware
      - phishing
    description: 'cryptowall C&C'

when i run command /opt/cif/bin/cif-smrt --testmode -v -c -d -r /etc/cif/rules/default/Cyber_Threat_Alliance.yml

cif@test:~$ /opt/cif/bin/cif-smrt --testmode -vvv -c -d -r /etc/cif/rules/default/Cyber_Threat_Alliance.yml -f CTA_cryptowall
[2015-11-09T15:41:49,559Z][4593][INFO][main:231]: staring up...
[2015-11-09T15:41:49,560Z][4593][DEBUG][main:288]: running pid: 4594
[2015-11-09T15:41:49,563Z][4594][DEBUG][main:370]: cleaning up tmp: /var/smrt/cache
[2015-11-09T15:41:49,565Z][4594][INFO][main:376]: removing: /var/smrt/cache/20151109.log
[2015-11-09T15:41:49,601Z][4594][INFO][main:307]: checking for router...
[2015-11-09T15:41:51,063Z][4594][DEBUG][main:321]: http://cyberthreatalliance.org - CTA_cryptowall
[2015-11-09T15:41:51,063Z][4594][INFO][main:323]: processing: -r /etc/cif/rules/default/Cyber_Threat_Alliance.yml -f CTA_cryptowall
[2015-11-09T15:41:51,064Z][4594][DEBUG][CIF::Smrt:99]: starting at: 2015-11-09T00:00:00Z
[2015-11-09T15:41:51,064Z][4594][DEBUG][CIF::Smrt:104]: fetching...
[2015-11-09T15:41:51,066Z][4594][DEBUG][CIF::Smrt:111]: cache: /var/smrt/cache/http:__cyberthreatalliance.org-CTA_cryptowall
[2015-11-09T15:41:51,066Z][4594][DEBUG][CIF::Smrt:114]: decoding..
[2015-11-09T15:41:51,068Z][4594][DEBUG][CIF::Smrt:84]: data is of type: application/octet-stream
[2015-11-09T15:41:51,069Z][4594][DEBUG][CIF::Smrt:118]: parsing...
[2015-11-09T15:41:51,070Z][4594][DEBUG][CIF::Smrt:124]: checking journal
[2015-11-09T15:41:51,073Z][4594][DEBUG][CIF::Smrt:230]: using log: /var/smrt/cache/20151109.log
[2015-11-09T15:41:51,085Z][4594][DEBUG][CIF::Smrt:127]: writing journal...
[2015-11-09T15:41:51,102Z][4594][DEBUG][CIF::Smrt:131]: processing events: 17
[2015-11-09T15:41:51,145Z][4594][INFO][CIF::Smrt:179]: processed events: 0
[2015-11-09T15:41:51,145Z][4594][INFO][main:359]: nothing new to send...
[2015-11-09T15:41:51,146Z][4594][INFO][main:363]: completed
[2015-11-09T15:41:51,203Z][4593][INFO][main:254]: shutting down...

cif-smrt doesnt parse it.

any idea?

[giovino]

We generally consider those types of feeds "overloaded". What is the feed author trying to communicate? If I want to try to use the observables in network sensors do I treat every observable with an equal weight? Typically that is not the case.

The easiest way to handle a feed like that is to have cif-smrt parse the feed multiple times parsing out a different observable on each pass.

You can seen an example of this in the zeus tracker feed:

https://github.com/csirtgadgets/massive-octo-spice/blob/develop/src/rules/default/zeustracker.yml

On the feed hxxps://zeustracker.abuse.ch/monitor.php?urlfeed=binaries...

1. On the first pass it grabs the urls
2. On the second pass it grabs the hash

This allows you to tag each observable type with completely different meta data (e.g. confidence) if you so choose.

Make sense?

[epazoglu]

Thanks Giovanni,

i did it as you said, date is not parsed so i remove date for test, and all observables parsed correctly.

Now i have 2 question : )

1- Is Date format must be YYYY-MM-DDTHH:MM:SSZ i ask because this feed's date format is different and i transform with sed and awk as

2015-4-21T8:35:00

and if Month,Day,Hour is not 2 digit (like 1-9) it doesnt add "0" before variable. Must i add "0" for this 1 digit variables ?

2- hash observable is parsed and i can see with cif client when i run

cif --today  --description cryptowall

and cif --today --description cryptowall --otype ipv4 cif --today --description cryptowall --otype fqdn also works but

cif --today --description cryptowall --otype hash

doesnt work.

How can i only pull hash observable ?

[giovino]

Q1:

I do not know if that date format is supported or not... off the top of my head. Testing is probably the easiest thing to do. CIF calls date-parse to try to auto figure out format it does not natively understand.

Q2:

I also could not get --otype hash to work, I would have guessed it was supported but we don't have any doc that says that it is. I've open an issue to have it looked into.

[giovino]

@evrenbey

Querying for a list of hashes needs to be done like this:

https://github.com/csirtgadgets/massive-octo-spice/wiki/Introducing-the-CIF-client#by-observable-type

I'm closing this issue, please let us know if you have further questions.