Open djaglowski opened 1 week ago
Pinging code owners:
pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley
See Adding Labels via Comments if you do not have permissions to add labels yourself.
@djaglowski we support map literals now, will that help? The grammar supports:
ParseSeverity(attributes["sev"], {"err": ["error", "NOOO"], "info": ["hey"]})
As a reference for discussion, the existing solution would be multiple statements with conditions:
set(severity_number, SEVERITY_NUMBER_ERROR) where attributes["sev"] == "error" or attributes["sev"] == "NOOO"
set(severity_number, SEVERITY_NUMBER_INFO) where attributes["sev"] == "hey" or (attributes["status code"] >= 1 and attributes["status code"] < 100)
Any additional function will need to be simpler than those statements.
It is possible that the solution doesn't need to be an additional function and instead could be a additional section in the transformprocessor config.
we support map literals now, will that help?
I don't think this is sufficient for ranges, which are a common use case with severity interpretation.
Any additional function will need to be simpler than those statements.
I don't think that's difficult but performance should also be a consideration here.
I'll give another example of statements which could be much clearer and more performant with a better solution.
In this case, HTTP status codes are interpreted into severity number and text:
set(severity_number, 5) where (IsMatch(body["status"], "^1[0-9]{2}$") and true)
set(severity_text, "debug") where (IsMatch(body["status"], "^1[0-9]{2}$") and true)
set(severity_number, 9) where (IsMatch(body["status"], "^2[0-9]{2}$") and true)
set(severity_text, "info") where (IsMatch(body["status"], "^2[0-9]{2}$") and true)
set(severity_number, 9) where (IsMatch(body["status"], "^3[0-9]{2}$") and true)
set(severity_text, "info") where (IsMatch(body["status"], "^3[0-9]{2}$") and true)
set(severity_number, 13) where (IsMatch(body["status"], "^4[0-9]{2}$") and true)
set(severity_text, "warn") where (IsMatch(body["status"], "^4[0-9]{2}$") and true)
set(severity_number, 17) where (IsMatch(body["status"], "^5[0-9]{2}$") and true)
set(severity_text, "error") where (IsMatch(body["status"], "^5[0-9]{2}$") and true)
This is both difficult to understand and not very performant since each statement must execute against each log record, and because the statements themselves are non-trivial to evaluate.
In contrast, the stanza is far simpler to understand:
severity:
parse_from: body["status"]
mapping:
debug:
- min: 100
max: 199
info:
- min: 200
max: 299
- min: 300
max: 399
warn:
- min: 400
max: 499
error:
- min: 500
max: 599
# (Actually it can be even simpler but this is a special case where aliases represent predefined ranges)
# mapping:
# debug: 1xx
# info:
# - 2xx
# - 3xx
# warn: 4xx
# error: 5xx
Additionally, there is a substantial performance optimization which occurs with the stanza configuration. Specifically, we're able to put all values into a map at startup, so that every log records requires exactly one map lookup.
I don't have strong opinions about exactly which form this should take in OTTL but I think it should be supported in some way that avoids such dense and wasteful statements.
Component(s)
pkg/ottl
Is your feature request related to a problem? Please describe.
OTTL can directly set severity number and text, but it would be nice if there was an optimized function for interpreting values. Stanza currently has a dedicated severity interpreter which has a couple advantages over using OTTL:
Describe the solution you'd like
The primary challenge with OTTL is that its functional nature may not lend itself well to specifying mappings. I'm not sure if there is a way to do this today, but I'd like users to be able to specify a function that contains an arbitrary number of mapping options. e.g.
ParseSeverity(attributes["sev"], AsError("err", "error", "NOOO"), AsInfo("info", "hey"), AsInfoRange(1, 100), ...)
Describe alternatives you've considered
No response
Additional context
No response