add commit/owner/repo/sentry context to all log messages

matt-codecov commented 4 days ago

currently worker logs have celery task information that you can query in GCP Logs Explorer with something like

jsonPayload.task_name="app.tasks.compute_comparison.ComputeComparison"
jsonPayload.task_id="c70bf145-dbcc-43ff-970d-dcf9251e289f"

those fields are still there, but this PR adds a lot more:

task info (new location) jsonPayload.context.task_name and jsonPayload.context.task_id
repo info jsonPayload.context.repo_name and jsonPayload.context.repo_id
owner info jsonPayload.context.owner_name, jsonPayload.context.owner_service, jsonPayload.context.owner_id
commit info jsonPayload.context.commit_sha and jsonPayload.context.commit_id
- note: in many places we call the SHA the commitid but here the SHA is commit_sha and the database ID is commit_id
sentry info jsonPayload.context.sentry_trace_id

some queries you'll be able to run in GCP Logs Explorer:

all logs for our worker repo:

jsonPayload.context.owner_service="github"
jsonPayload.context.owner_username="codecov"
jsonPayload.context.repo_name="worker"

all logs for a commit across tasks:

-- technically you should also include the repo/owner in case of mirrors/forks
jsonPayload.context.commit_sha="186d5c6f82b0ab4dac20d276f2cc56113099a649"

all logs for a sentry trace (including API logs that have been tagged!)

jsonPayload.context.sentry_trace_id="" OR jsonPayload.sentry_trace_id="31cfa9b2c81e41b6a662f9de639d80b6"

some fields will be nulled out. for example SyncRepos is an owner-scoped operation so it will have owner info but not repo/commit info. but uploads are commit-specific so Upload should have commit, repo, and owner info.

we log URLs that include owner/repo names in many places so adding them to the auto-populated log context doesn't introduce any new concerns

codecov[bot] commented 4 days ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 97.99%. Comparing base (6b1f38e) to head (0659eb5). Report is 6 commits behind head on main.

:white_check_mark: All tests successful. No failed tests found.

Additional details and impacted files

[![Impacted file tree graph](https://app.codecov.io/gh/codecov/worker/pull/813/graphs/tree.svg?width=650&height=150&src=pr&token=BWTOrjBaE5&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov)](https://app.codecov.io/gh/codecov/worker/pull/813?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov) ```diff @@ Coverage Diff @@ ## main #813 +/- ## ======================================= Coverage 97.99% 97.99% ======================================= Files 446 446 Lines 36568 36621 +53 ======================================= + Hits 35835 35888 +53 Misses 733 733 ``` | [Flag](https://app.codecov.io/gh/codecov/worker/pull/813/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov) | Coverage Δ | | |---|---|---| | [integration](https://app.codecov.io/gh/codecov/worker/pull/813/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov) | `97.99% <100.00%> (+<0.01%)` | :arrow_up: | | [unit](https://app.codecov.io/gh/codecov/worker/pull/813/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov) | `97.99% <100.00%> (+<0.01%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov#carryforward-flags-in-the-pull-request-comment) to find out more. | [Components](https://app.codecov.io/gh/codecov/worker/pull/813/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov) | Coverage Δ | | |---|---|---| | [NonTestCode](https://app.codecov.io/gh/codecov/worker/pull/813/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov) | `95.85% <100.00%> (+0.01%)` | :arrow_up: | | [OutsideTasks](https://app.codecov.io/gh/codecov/worker/pull/813/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov) | `97.97% <100.00%> (+<0.01%)` | :arrow_up: | | [Files with missing lines](https://app.codecov.io/gh/codecov/worker/pull/813?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov) | Coverage Δ | | |---|---|---| | [helpers/log\_context.py](https://app.codecov.io/gh/codecov/worker/pull/813?src=pr&el=tree&filepath=helpers%2Flog_context.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov#diff-aGVscGVycy9sb2dfY29udGV4dC5weQ==) | `98.61% <100.00%> (+1.46%)` | :arrow_up: | | [helpers/logging\_config.py](https://app.codecov.io/gh/codecov/worker/pull/813?src=pr&el=tree&filepath=helpers%2Flogging_config.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov#diff-aGVscGVycy9sb2dnaW5nX2NvbmZpZy5weQ==) | `82.22% <100.00%> (+0.40%)` | :arrow_up: | | [helpers/tests/unit/test\_log\_context.py](https://app.codecov.io/gh/codecov/worker/pull/813?src=pr&el=tree&filepath=helpers%2Ftests%2Funit%2Ftest_log_context.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov#diff-aGVscGVycy90ZXN0cy91bml0L3Rlc3RfbG9nX2NvbnRleHQucHk=) | `100.00% <100.00%> (ø)` | | | [helpers/tests/unit/test\_logging\_config.py](https://app.codecov.io/gh/codecov/worker/pull/813?src=pr&el=tree&filepath=helpers%2Ftests%2Funit%2Ftest_logging_config.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=codecov#diff-aGVscGVycy90ZXN0cy91bml0L3Rlc3RfbG9nZ2luZ19jb25maWcucHk=) | `100.00% <100.00%> (ø)` | |

codecov-notifications[bot] commented 4 days ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

:white_check_mark: All tests successful. No failed tests found.

@@           Coverage Diff           @@
##             main     #813   +/-   ##
=======================================
  Coverage   97.99%   97.99%           
=======================================
  Files         446      446           
  Lines       36568    36621   +53     
=======================================
+ Hits        35835    35888   +53     
  Misses        733      733

Flag	Coverage Δ
integration	`97.99% <100.00%> (+<0.01%)`	:arrow_up:
unit	`97.99% <100.00%> (+<0.01%)`	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
NonTestCode	`95.85% <100.00%> (+0.01%)`	:arrow_up:
OutsideTasks	`97.97% <100.00%> (+<0.01%)`	:arrow_up:

Files with missing lines	Coverage Δ
helpers/log_context.py	`98.61% <100.00%> (+1.46%)`	:arrow_up:
helpers/logging_config.py	`82.22% <100.00%> (+0.40%)`	:arrow_up:
helpers/tests/unit/test_log_context.py	`100.00% <100.00%> (ø)`
helpers/tests/unit/test_logging_config.py	`100.00% <100.00%> (ø)`

codecov-qa[bot] commented 4 days ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 97.99%. Comparing base (6b1f38e) to head (0659eb5). Report is 6 commits behind head on main.

:white_check_mark: All tests successful. No failed tests found.

@@           Coverage Diff           @@
##             main     #813   +/-   ##
=======================================
  Coverage   97.99%   97.99%           
=======================================
  Files         446      446           
  Lines       36568    36621   +53     
=======================================
+ Hits        35835    35888   +53     
  Misses        733      733

Flag	Coverage Δ
integration	`97.99% <100.00%> (+<0.01%)`	:arrow_up:
unit	`97.99% <100.00%> (+<0.01%)`	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
NonTestCode	`95.85% <100.00%> (+0.01%)`	:arrow_up:
OutsideTasks	`97.97% <100.00%> (+<0.01%)`	:arrow_up:

Files with missing lines	Coverage Δ
helpers/log_context.py	`98.61% <100.00%> (+1.46%)`	:arrow_up:
helpers/logging_config.py	`82.22% <100.00%> (+0.40%)`	:arrow_up:
helpers/tests/unit/test_log_context.py	`100.00% <100.00%> (ø)`
helpers/tests/unit/test_logging_config.py	`100.00% <100.00%> (ø)`

codecov-public-qa[bot] commented 4 days ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 97.99%. Comparing base (6b1f38e) to head (0659eb5).

:white_check_mark: All tests successful. No failed tests found.

@@           Coverage Diff           @@
##             main     #813   +/-   ##
=======================================
  Coverage   97.99%   97.99%           
=======================================
  Files         446      446           
  Lines       36568    36621   +53     
=======================================
+ Hits        35835    35888   +53     
  Misses        733      733

Flag	Coverage Δ
integration	`97.99% <100.00%> (+<0.01%)`	:arrow_up:
unit	`97.99% <100.00%> (+<0.01%)`	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
NonTestCode	`95.85% <100.00%> (+0.01%)`	:arrow_up:
OutsideTasks	`97.97% <100.00%> (+<0.01%)`	:arrow_up:

Files	Coverage Δ
helpers/log_context.py	`98.61% <100.00%> (+1.46%)`	:arrow_up:
helpers/logging_config.py	`82.22% <100.00%> (+0.40%)`	:arrow_up:
helpers/tests/unit/test_log_context.py	`100.00% <100.00%> (ø)`
helpers/tests/unit/test_logging_config.py	`100.00% <100.00%> (ø)`

codecov-test commented 4 days ago

Codecov Report

Attention: Patch coverage is 43.07692% with 37 lines in your changes missing coverage. Please review.

Project coverage is 97.27%. Comparing base (6b1f38e) to head (0659eb5).

Files with missing lines	Patch %	Lines
helpers/log_context.py	45.45%	24 Missing :warning:
helpers/tests/unit/test_log_context.py	26.66%	11 Missing :warning:
helpers/logging_config.py	0.00%	1 Missing :warning:
helpers/tests/unit/test_logging_config.py	80.00%	1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

@@            Coverage Diff             @@
##             main     #813      +/-   ##
==========================================
- Coverage   97.40%   97.27%   -0.13%     
==========================================
  Files         441      441              
  Lines       36564    36576      +12     
==========================================
- Hits        35614    35581      -33     
- Misses        950      995      +45

Components	Coverage Δ
NonTestCode	`94.85% <44.44%> (-0.07%)`	:arrow_down:
OutsideTasks	`97.48% <43.07%> (-0.15%)`	:arrow_down:

Files with missing lines	Coverage Δ
helpers/logging_config.py	`53.33% <0.00%> (+0.30%)`	:arrow_up:
helpers/tests/unit/test_logging_config.py	`83.87% <80.00%> (-16.13%)`	:arrow_down:
helpers/tests/unit/test_log_context.py	`56.52% <26.66%> (-43.48%)`	:arrow_down:
helpers/log_context.py	`37.50% <45.45%> (-13.82%)`	:arrow_down:

... and 1 file with indirect coverage changes

codecov / worker