feat(observability): OpenTelemetry logs

samugi commented 3 months ago

Note for reviewers: commits are split and commented appropriately, I would recommend checking them one by one.

Summary

This PR adds OpenTelemetry formatted logs as a new signal supported by the OpenTelemetry plugin.

It patches the ngx.log function (and adapts kong PDK log) to collect the logged lines and enrich them with additional context info:

severity
introspection details
timestamp

And for request-scoped logs also:

Request ID
Trace ID
Span ID
Trace flags

The log entries are protobuf-serialized and sent to the endpoint configured in the OpenTelemetry plugin's config.logs_endpoint field.

Checklist

[x] The Pull Request has tests
[x] A changelog file has been added to CHANGELOG/unreleased/kong or adding skip-changelog label on PR if unnecessary. README.md
[x] The Pull Request has backports to all the versions it needs to cover
[x] There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - https://github.com/Kong/docs.konghq.com/pull/7583

Issue reference

KAG-4712

samugi commented 3 months ago

And also I suggest not rebase commits when applying reviewing suggestions, we can use the following command for better view for the reviewer.
git commit --fixup <commit_id>

yes absolutely. I wasn't planning to rebase (for now), which will make syncing (via cherry-pick) with the existing EE PR easier as well.

ADD-SP commented 3 months ago

And also I suggest not rebase commits when applying reviewing suggestions, we can use the following command for better view for the reviewer.
git commit --fixup <commit_id>
absolutely. I wasn't planning to rebase, which will make syncing (via cherry-pick) with the existing EE PR easier as well.

https://thoughtbot.com/blog/autosquashing-git-commits

Not sure if you have read this article, hope this can help you.

samugi commented 3 months ago

no logic changes in recent commits, I've only fixed compat with older DPs and added tests.

Achiel commented 3 months ago

@ADD-SP if possible could we get this reviewed on Monday? We have a wonderful set of customers that I'd love to show this off to as a beta. Thanks!

team-gateway-bot commented 2 months ago

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-13291-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-13291-to-master-to-upstream
git checkout -b cherry-pick-13291-to-master-to-upstream
ancref=$(git merge-base 8f9b82d870985cdd5fb417d589ad2f7ddec967e4 c1f99a9ff2a5986dbd72e85d9897167df2124b77)
git cherry-pick -x $ancref..c1f99a9ff2a5986dbd72e85d9897167df2124b77

Kong / kong