search

HTTPArchive / bigquery

BigQuery import and processing pipelines
67 stars 20 forks source link

Add ranking lenses #121

Closed tunetheweb closed 3 years ago

tunetheweb commented 3 years ago

Makes progress on https://github.com/HTTPArchive/httparchive.org/issues/377

Closes #49 by removing generate_report.sh as now no longer needed and missing some of the more recent changes.

rviscomi commented 3 years ago

Only added top1k and top10k lenses. Not sure we need them all. WDYT?

I think it's worth adding all of them to encourage exploration of trends in the torso/tail.

tunetheweb commented 3 years ago

Yeah I changed my mind and added the rest as only two more (as don’t need last one) but forgot to update the initial comment.

tunetheweb commented 3 years ago

OK I switched to this branch and ran the top10k lens (which took a few attempts with various fixes) and then the top100k lens (which worked first time!).

So think this is good to merge. Will do that unless you have any other comments @rviscomi or wanna spend more time looking at it?

Will run the other two lenses (top1k, and top1m) after merging, then keep a close eye on it next month.