Allow Heroku logs containing dyno metrics to be transformed into useable attributes

[jdeff]

Component(s)

OTTL

Is your feature request related to a problem? Please describe.

Heroku emits log lines containing metrics data for the various dyno instances running for an application. These log lines look similar to:

source=web.1 dyno=heroku.1234-5678-9012-3456 sample#memory_total=10917.88MB sample#memory_rss=10849.69MB sample#memory_cache=68.20MB sample#memory_swap=0.00MB sample#memory_pgpgin=3384pages sample#memory_pgpgout=82pages sample#memory_quota=14336.00MB

Using the syslog_receiver and key_value_parser you can get something almost useable. However, the values are still strings of human readable byte sizes (e.g. "68.20MB") and need to converted to an integer number of bytes in order for them to be useable.

Describe the solution you'd like

I would like to be able to parse these byte size values using the transformprocessor, which I am already using to shape other values, and convert them to an integer value of bytes. I propose adding a ParseBytes function to OTTL in the same vain as ParseJson and ParseCsv. This could be used with the following config:

processors:
    transform:
      log_statements:
        - context: log
          statements:
            - replace_all_patterns(attributes, "key", "sample#memory-total", "system.memory.usage.used")
            - set(attributes["system.memory.usage.used"], ParseBytes(attributes["system.memory.usage.used"])) where attributes["system.memory.usage.used"] != nil

Describe alternatives you've considered

I've looked at simply striping the units from these values, but the units are not guaranteed to be the same.

ParseBytes may be an ambiguous or confusing name (though it is what is used by go-humanize). Something like ParsePrettyByteSize or ParseHumanizedByteString might be clearer.

Additional context

No response

[jdeff]

cc @TylerHelmuth

[github-actions[bot]]

Pinging code owners for processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions[bot]]

Pinging code owners for pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

Do you want those logs to be transformed into metric data?

[TylerHelmuth]

The ability to convert regular units to another unit is generally useful. I think a more generic function that takes an input and converts to a desired output would be most generally useful.

Warning, there are some weird edge cases with converting when you need to support stuff like mb, M, and Mi. It was so annoying we ended up writing a solution ourselves instead of using a library

[TylerHelmuth]

@jdeff can you propose what a function would look like to be able to convert 10917.88MB to any format?