search

mozilla / identity-ops

Tools and Chef cookbooks used by Mozilla Services Operations to provision and manage Persona
Other
24 stars 12 forks source link

count unique RPs for a 1 day period #119

Closed kparlante closed 11 years ago

kparlante commented 11 years ago

@gene1wood Here's the line grep "GET /sign_in" logfile | awk '{print $14}' | sort | uniq -c > output.txt

I might not have the url quite right -- the sample you gave me had sign_in as the referrer -- we're looking to count unique referrers for calls to sign_in

We want to run this on log files from all regions in a given time period -- the resulting output.txt should be small.

gene1wood commented 11 years ago

Hmm, looks like field 14 is inside the user agent

zcat /var/log/nginx/access.log-20130818.gz | grep "GET /sign_in" | awk '{print $14}' | head
NT
NT
NT
NT
NT
NT
Intel
NT
NT
NT

What field are you looking to grab, the referrer?

kparlante commented 11 years ago

oops, I copied crap out of the adium chat into my sample file. Yes, the referrer, should be $11 I think

gene1wood commented 11 years ago

@kparlante And are you looking for a list of all referrer URLs ordered by count or for referrer sites ordered by count ( http://mineshafter.info/downloads vs mineshafter.info )

kparlante commented 11 years ago

Yeah, sites not urls (That shouldn't blow up the file sizes too much though -- I can process that out when combining the results from the different regions)

gene1wood commented 11 years ago

I've emailed the results to you