search

dannycoates / smt

silly metrics tools
0 stars 0 forks source link

Is this mozlog format ok for metrics? #3

Open mostlygeek opened 7 years ago

mostlygeek commented 7 years ago

In https://github.com/mozilla-services/go-syncstorage/issues/136 I refactored the output to look like this: (PII has been scrubbed, but it'll log the fxa_uid and device_id from the tokenserver)

{"Timestamp":1475710133088049792,"Type":"request.summary","Logger":"go-syncstorage","Hostname":"ip-172-31-25-181","EnvVersion":"2.0","Pid":1,"Severity":6,"Fields":{"agent":"Firefox/49.0 (Linux x86_64) FxSync/1.51.0.20160923002641.desktop","device_id":"scrub","errno":0,"fxa_uid":"scrub","method":"POST","path":"/1.5/scrub/storage/prefs","req_sz":11179,"res_sz":106,"t":7,"uid":"scrub"}}
{"Timestamp":1475710133089309333,"Type":"request.summary","Logger":"go-syncstorage","Hostname":"ip-172-31-25-181","EnvVersion":"2.0","Pid":1,"Severity":6,"Fields":{"agent":"Firefox/49.0 (Linux x86_64) FxSync/1.51.0.20160920074044.desktop","device_id":"scrub","errno":0,"fxa_uid":"scrub","method":"GET","path":"/1.5/scrub/info/collections","req_sz":0,"res_sz":207,"t":0,"uid":"scrub"}}
{"Timestamp":1475710133137815436,"Type":"request.summary","Logger":"go-syncstorage","Hostname":"ip-172-31-25-181","EnvVersion":"2.0","Pid":1,"Severity":6,"Fields":{"agent":"Firefox/49.0.1 (Windows NT 10.0) FxSync/1.51.0.20160922113459.desktop","device_id":"scrub","errno":0,"fxa_uid":"scrub","method":"GET","path":"/1.5/scrub/storage/tabs?newer=1475710040.56\u0026full=1","req_sz":0,"res_sz":17393,"t":0,"uid":"scrub"}}
{"Timestamp":1475710133144862391,"Type":"request.summary","Logger":"go-syncstorage","Hostname":"ip-172-31-25-181","EnvVersion":"2.0","Pid":1,"Severity":6,"Fields":{"agent":"Firefox/49.0.1 (Windows NT 10.0; WOW64) FxSync/1.51.0.20160922113459.desktop","device_id":"scrub","errno":0,"fxa_uid":"scrub","method":"POST","path":"/1.5/scrub/storage/tabs","req_sz":6864,"res_sz":66,"t":10,"uid":"scrub"}}
{"Timestamp":1475710133239684942,"Type":"request.summary","Logger":"go-syncstorage","Hostname":"ip-172-31-25-181","EnvVersion":"2.0","Pid":1,"Severity":6,"Fields":{"agent":"Firefox/49.0 (Intel Mac OS X 10.11) FxSync/1.51.0.20160916101415.desktop","device_id":"scrub","errno":0,"fxa_uid":"scrub","method":"POST","path":"/1.5/scrub/storage/prefs","req_sz":15019,"res_sz":106,"t":7,"uid":"scrub"}}

Are the above mozlog formatted logs ok? @whd @dannycoates ?

whd commented 7 years ago

Looks ok. Things to note:

From my cursory inspection you've added all the changes in PRs to this repo for naming differences between go mozlog and the old format already.

The Type should be mozsvc.metrics if possible, as that is what we match on to determine it is a log that we actually care about for analysis (there are other types such as mozsvc.user in the old logging).

I don't see a uid field in the current sync app logs, so possibly that is unnecessary. An example line (in the old format): 

{"code": 200, "v": 1, "pid": 0, "agent": "Firefox/49.0 (Linux x86_64) FxSync/1.51.0.20160920074044.desktop", "fxa_uid": "scrub", "path": "https://sync-XXX-us-west-2.sync.services.mozilla.com/1.5/scrub/storage/meta/global", "syncstorage.storage.sql.db.execute": 0.00, "syncstorage.storage.sql.pool.get": 0.00, "remoteAddressChain": ["scrub", "scrub"], "request_time": 0.00, "hostname": "ip-000-00-00-000", "name": "mozsvc.metrics", "time": "2016-10-06T00:00:00.0000Z", "device_id": "scrub", "method": "GET", "op": "mozsvc.metrics"}

I'm going to need to change the aggregator Logger matching to include go-syncstorage in addition to Sync-1_5 to pick these up.

mostlygeek commented 7 years ago

Thanks for the feedback. I changed the Type and Logger to match the python sync 1.5