Utility for filtering fields at the visitor level

[hawkw]

Feature Request

Crates

• tokio-trace-subscriber
• possibly tokio-trace-fmt

Motivation

Spans and events may be created with up to 32 fields. In many cases, these fields will provide fairly verbose diagnostic information about the program. Sometimes, users may wish to record data about a span or event, but exclude particularly verbose data attached to it (such as fields provided by libraries).

For example, tokio-trace-futures currently adds fields describing the task to spans generated by the spawn! macro. In the future, the tokio runtime will likely enrich these spans with additional information about the task's state in the executor. It's likely that this additional information will only be necessary when diagnosing issues involving the executor or the application's interaction with it, so users probably will not typically want to display them by default. Fortunately, the introduction of . into the permissible field name characters allows "namespacing" these fields; for example, task-specific data is currently in fields with names such as tokio.task.$FIELD.

Proposal

I propose adding a utility for wrapping a Visit implementation with a FieldFilter (or similar) struct. This wrapper would implement Visit by checking if field names match a filter, and if they do not, it would do nothing. Otherwise, it would forward to the wrapped visitor. This could live in tokio-trace-subscriber, as it's primarily a tool for subscriber implementors to use.

Additionally, we'd have functions for constructing a "field filter" from various types, such as a Fn(&Field) -> bool, and a list of allowed or denied patterns.

As a proposal, I suggest a grammar for constructing allow/deny lists from strings similar to the EnvFilter in tokio-trace-fmt. A single filter pattern would consist of:

• a field name: matches that field name exactly.
  • for example, tokio.task.is_spawned, message, user.email
• single globs: a single asterisk * should match up to the next . character.
  • for example:
    • tokio.task.* would match any field name beginning with tokio.task. followed by one more Rust identifier
    • tokio.*.is_spawned: matches any field name beginning with tokio., followed by a single rust identifier segment, followed by .is_spawned
    • tokio.task.is_*: matches tokio.task.is_spawned, tokio.task.is_blocking, tokio.task.is_parked, and so on.
    • *.*.* matches any field name consisting of exactly three dotted segments
• double globs: two asterisks (**) would match any number of characters.
  • for example:
    • tokio.** would match tokio.task.is_spawned,tokio.runtime, tokio.task.state.is_blocked, and so on
    • **.is_spawned: matches tokio.task.is_spawned, romio.is_spawned, foo.executor.state.is_spawned, std.thread.is_spawned, and so on
    • **: matches anything

We could parse an allow/deny list from a comma separated list of the above patterns.

Additionally, we could add syntax to filters that are dependent on the name of the span/event the field is on, as well as the field name itself. I think that should probably be a follow-up issue, since it requires an API for informing the visitor what the span's name is.

Alternatives

Rather than implementing our own globbing syntax, we could use regular expressions. This might be simpler to implement, since we can use existing regex libraries, and would allow more flexible filters.

However, since we have a much more restricted grammar for field names than "all UTF-8 characters", there are advantages for not allowing full regular expressions. For example, since , is not a valid field name character, we can parse a list of patterns from a comma-separated string. However, if we accept arbitrary regexen, , may be part of a regex, rather than a separator between regexen.

Similarly, . is a special character with a defined meaning in most regex implementations. This means that regexen that match tokio-trace field names with .s would need to escape every . character. This would lead to regexen like foo\..+.bar, which is awkward and hard to parse.

Finally, it would be possible to construct a regex that matches only field names that are invalid and will never occur. Allowing such a regex and testing all field names against it seems inefficient. Using our own globbing syntax would allow us to reject patterns that will never match a field.

[hawkw]

241 has been merged, which should hopefully unblock this.

[dzfranklin]

I'm working on a draft pr in #1345 that addresses this.

[torsteingrindvik]

Adding a "user story" comment, since I think I am looking for this feature.

I log to stdout, file, and OpenTelemetry.

Because I want the OpenTelemetry frontend (jaeger) to display things correctly, I add an extra field to my span: otel.name = %var.

But this looks ugly in stdout and file logs, so I would like to not display fields starting with otel.* there (and not implement a whole new formatter or similar).

So I guess +1 to this feature.

[garypen]

Similar use case to @torsteingrindvik (otel.*). It would be nice to have more control over the span fields which are logged in the Full formatter. That's another +1 for this feature.