Getting real TG data in ES/Kibana

[nrezvani]

We need to get real TG data in ES and Kibana.

Proposed discussed strategy: We can run the existing migration for importing TG feeds for a start and then also using that same code to generate sightings. Mainly, we can

1. Take the existing feed-> judgement code and update it if needed.
2. Run it on a month worth of data.
3. Modify it to generate a sighting record (since a feed hit is also a sighting of an indicator...)
4. Load them all

We should also:

1. Create Indicator objects for the top level "feeds" so each feed is an indicator, e.g., Banking Trojan, RAT DNS, etc.

[saintx]

For a daily dump of a feed, use the REST API. Example: https://sandcastle.threatgrid.com/api/v3/feeds/rat-dns_2016-02-02.json?api_key=KEY

List of available feeds:

• autorun-registry
• banking-dns
• dll-hijacking-dns
• doc-net-com-dns
• downloaded-pe-dns
• dynamic-dns
• irc-dns
• modified-hosts-dns
• parked-dns
• public-ip-check-dns
• rat-dns
• scheduled-tasks
• sinkholed-ip-dns
• stolen-cert-dns

[saintx]

• [ ] Create Indicator objects for the top level "feedsf so each feed is an Indicator
• [ ] Generate Judgements from feed events
• [ ] Generate Sightings from feed events