search

mozkeeler / sunlight

Examine the Certificate Transparency Log for Baseline Requirements issues
MIT License
2 stars 1 forks source link

work on #14: initial dashboard implemetation #48

Closed monicachew closed 9 years ago

monicachew commented 9 years ago

Getting this into a PR so it's easier to find.

monicachew commented 9 years ago

I worked on this a little bit then decided that it's better to preprocess the data from the DB -- since the json is 5M it doesn't make sense for the client to do all the work. But that's not very important perhaps.

mozkeeler commented 9 years ago

Sure - I guess we can extend db2json.js to process/format the data we want? Eventually I was thinking the graphs would be dynamic and basically be a front-end onto the DB, but that's farther down the line (and could maybe be a different view or product/project altogether).

monicachew commented 9 years ago

I think a reasonable workflow would be a cronjob to update the CT DB every day or so, run the go script, then dump the DB in the dashboard directory.

monicachew commented 9 years ago

@mozkeeler if you're OK with this approach, I can go ahead and dump all timeseries for all "interesting" issuers programmatically from the node file. Strangely the beginTime times don't seem to be on month boundaries...

monicachew commented 9 years ago

Also the post-processed file is around 1.5M now -- but we can load those dynamically depending on what the user wants to see.

mozkeeler commented 9 years ago

Sounds good to me!

monicachew commented 9 years ago

This is now up at http://people.mozilla.org/~mchew/dashboard/. I omitted the validPeriodTooLong because the casing was incorrect in the go tool, but I can put that back after I re-run it.

Things to do: 1) Probably dump to different files, the output of db2json is now back up to 6.9M and we haven't dumped the series for "worst" offenders yet. 2) Figure out if/how we want to switch between raw/normalized scores. 3) Figure out the timestamp truncation issue, supposedly issuerReputation should only have beginTimes at the start of the month.

mozkeeler commented 9 years ago

We probably don't need dashboard/issuerReputation.json now, right?

monicachew commented 9 years ago

Probably not. That whole change needs cleanup, there are some bad names. Also we may need a sidebar(?) or something similar to toggle between stuff.

On Wed, Jan 7, 2015 at 1:44 PM, David Keeler notifications@github.com wrote:

We probably don't need dashboard/issuerReputation.json now, right?

— Reply to this email directly or view it on GitHub https://github.com/mozkeeler/sunlight/pull/48#issuecomment-69097672.

mozkeeler commented 9 years ago

Right - I was thinking the sidebar could be a next step once we've landed this initial work. Do you want to do the cleanup or should I go ahead?

monicachew commented 9 years ago

Please go ahead, I'm not sure I have time to work on this until next week.

On Wed, Jan 7, 2015 at 1:58 PM, David Keeler notifications@github.com wrote:

Right - I was thinking the sidebar could be a next step once we've landed this initial work. Do you want to do the cleanup or should I go ahead?

— Reply to this email directly or view it on GitHub https://github.com/mozkeeler/sunlight/pull/48#issuecomment-69099859.

mozkeeler commented 9 years ago

This works for me. I think the dates issue was fixed in c1f467f5 - try rebuilding sunlight.go & tools/sunlight.go and re-generating the DB? In the meantime, I'm going to merge this so there's something to work off of in terms of smaller improvements.