search

mozilla / gcp-ingestion

Documentation and implementation of telemetry ingestion on Google Cloud Platform
https://mozilla.github.io/gcp-ingestion/
Mozilla Public License 2.0
75 stars 31 forks source link

DENG-1320 - Add support for decoding geo and user agent metadata in Cloud Logging-based telemetry #2437

Closed akkomar closed 1 year ago

akkomar commented 1 year ago

Goes together with https://github.com/mozilla/glean_parser/pull/612

codecov-commenter commented 1 year ago

Codecov Report

Patch coverage: 60.52% and project coverage change: -0.19% :warning:

Comparison is base (baefb57) 86.05% compared to head (a465ea8) 85.87%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2437 +/- ## ============================================ - Coverage 86.05% 85.87% -0.19% - Complexity 871 875 +4 ============================================ Files 122 123 +1 Lines 5084 5118 +34 Branches 505 509 +4 ============================================ + Hits 4375 4395 +20 - Misses 551 560 +9 - Partials 158 163 +5 ``` | Flag | Coverage Δ | | |---|---|---| | ingestion_beam | `82.30% <60.52%> (-0.20%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files Changed](https://app.codecov.io/gh/mozilla/gcp-ingestion/pull/2437?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla) | Coverage Δ | | |---|---|---| | [...m/src/main/java/com/mozilla/telemetry/Decoder.java](https://app.codecov.io/gh/mozilla/gcp-ingestion/pull/2437?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla#diff-aW5nZXN0aW9uLWJlYW0vc3JjL21haW4vamF2YS9jb20vbW96aWxsYS90ZWxlbWV0cnkvRGVjb2Rlci5qYXZh) | `88.88% <16.66%> (-6.35%)` | :arrow_down: | | [...zilla/telemetry/decoder/ExtractIpFromLogEntry.java](https://app.codecov.io/gh/mozilla/gcp-ingestion/pull/2437?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla#diff-aW5nZXN0aW9uLWJlYW0vc3JjL21haW4vamF2YS9jb20vbW96aWxsYS90ZWxlbWV0cnkvZGVjb2Rlci9FeHRyYWN0SXBGcm9tTG9nRW50cnkuamF2YQ==) | `67.85% <67.85%> (ø)` | | | [...a/com/mozilla/telemetry/decoder/ParseLogEntry.java](https://app.codecov.io/gh/mozilla/gcp-ingestion/pull/2437?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla#diff-aW5nZXN0aW9uLWJlYW0vc3JjL21haW4vamF2YS9jb20vbW96aWxsYS90ZWxlbWV0cnkvZGVjb2Rlci9QYXJzZUxvZ0VudHJ5LmphdmE=) | `63.88% <75.00%> (+0.25%)` | :arrow_up: | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/mozilla/gcp-ingestion/pull/2437/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

akkomar commented 1 year ago

I'm a little hazy on what the specific guarantees about log formatting are going to be present for server application logs. Stuff like whether we know which IP address related fields will have values and that they will have consistent meanings across applications (e.g. if there are levels of indirection in services handling and emitting logs, nginx proxies etc.).

We're aiming to ensure formatting and field meaning through clients using glean_parser-generated code for logging.