mlsecproject / combine

Tool to gather Threat Intelligence indicators from publicly available sources
https://www.mlsecproject.org/
GNU General Public License v3.0
655 stars 171 forks source link

Make sure we can export in a dir structure that tiq-test can handle #29

Closed alexcpsec closed 10 years ago

alexcpsec commented 10 years ago

MOAR work!

Here is how things look on the tiq-test data directory right now:

aperture-2:data alexcp$ ls
enriched    population  raw
aperture-2:data alexcp$ ls raw
public_inbound  public_outbound
aperture-2:data alexcp$ ls raw/pu
public_inbound/  public_outbound/
aperture-2:data alexcp$ ls raw/public_inbound/
20140615.csv.gz 20140618.csv.gz 20140622.csv.gz 20140625.csv.gz 20140628.csv.gz 20140701.csv.gz 20140704.csv.gz 20140707.csv.gz 20140710.csv.gz 20140713.csv.gz
20140616.csv.gz 20140619.csv.gz 20140623.csv.gz 20140626.csv.gz 20140629.csv.gz 20140702.csv.gz 20140705.csv.gz 20140708.csv.gz 20140711.csv.gz 20140714.csv.gz
20140617.csv.gz 20140620.csv.gz 20140624.csv.gz 20140627.csv.gz 20140630.csv.gz 20140703.csv.gz 20140706.csv.gz 20140709.csv.gz 20140712.csv.gz 20140715.csv.gz

Basically we have the following structure: data/[DATATYPE]/[DATAGROUP]/[YYYYMMDD].csv.gz considering that:

Please note the CSVs are gzipped. The code expects that as well.

krmaxwell commented 10 years ago

Going to implement this as a helper script, I think, rather than directly in the core.

krmaxwell commented 10 years ago

This is the only remaining release blocker I think. :happykyle: