First sample of clean disaggregated data in bitstore

[rufuspollock]

• Location: private.bits.cybergreen.net/clean/ as per structure agreed in #7
• Tabular Data Package per risk with {risk-id}.csv
  • Format as per tech architecture document (TODO document this in a repo)
• Prefer smaller “sample” dataset atm - so happy with 1:10 sampling

Planned due date: Tues 30th 6pm GMT

@chorsley @kxyne creating out of our call earlier today.

[rufuspollock]

@chorsley @kxyne did you see this?

[chorsley]

@rgrp It's probably not going to be today as we've been looking at the date issues as discussed, but it's on the schedule for tomorrow.

[rufuspollock]

@chorsley any update?

[chorsley]

The data is now available in private-bits-cybergreen-net under /dev/clean/dns-scan. This is 1:10 sampled, enriched data for 3 weeks in August, format is:

timestamp, risk id, IP address, ASN, country code.

[chorsley]

There is now a second risk type, ntp, under /dev/clean/ntp-scan. Additionally, we now have /dev/clean/risk_ids.json which explains how the numeric risk IDs map to string identifiers.

Please also note - it's highly likely we'll review these path names, so please treat these as "yet to be confirmed".

[rufuspollock]

@chorsley couple of quick comments based on use:

• Can we get header line in the CSV files by default i.e. first line is column headings
• Can the date field just be data or do we need time e.g. 2016-08-05 vs 2016-08-05 02:00:06.0+00?
• If we compress can we put compression as last part of file name e.g. xyz.csv.gz rather than xyz.csv.gz
• Generally you don't need to sort data export by time - though fine if you do. May be easier for you do dumps without worrying about order :-)

[chorsley]

@rgrp follow ups:

Can we get header line in the CSV files by default i.e. first line is column headings

We will make sure the ETLv2 produces this. I'll do that if / when we do another run with ETLv1.

Can the date field just be data or do we need time e.g. 2016-08-05 vs 2016-08-05 02:00:06.0+00?

There's likely some interesting analysis we can do here, like working out the distribution of a scan over time, or calculating biases towards a particular time of day. So, I'd prefer to keep this in.

If we compress can we put compression as last part of file name e.g. xyz.csv.gz rather than xyz.csv.gz

Yes, this was a quick hack on my part - I just tacked the .csv extension after the original file name, but you'll see it's not compressed. ETLv2 won't have this problem.

Generally you don't need to sort data export by time - though fine if you do. May be easier for you do dumps without worrying about order :-)

The scan data we receive for open NTP and DNS already roughly orders by date (I'm assuming it's the order the results are received - you can get the odd occasion where lines are out of order). In the enriched files you have, no extra sorting has been performed by us.

[rufuspollock]

To complete what remains:

• [x] All 5 types of data in there
• [x] Tidying up structure / naming as per previous comment
• [x] datapackage.json in place (and auto-generated)

[chorsley]

Just to update: seems like the CSV format is acceptable, so now working on getting all CSV sets up on S3 using ETLv2.

[rufuspollock]

@chorsley great! When do you think we'll have complete data in there?

[chorsley]

First 4 sets there, some glitches with NTP data sizes, but complete for the most part. I'll raise a new issue for NTP data problem.