Enrich the structure of traces emitted by Tokio

[hds]

Is your feature request related to a problem? Please describe. The traces produced by Tokio are great, however there are some areas in which the hierarchy in which the traces are emitted could be improved or doesn't make sense.

Specific issues are:

• No division by runtime (it isn't possible to distinguish which tasks belong to which runtime when there are multiple runtimes)
  • This would enable new Console features like tokio-rs/console#130, tokio-rs/console#229
  • Would be nice to have externally available runtime ID and name too (#5545).
• There is no information about runtimes at all
  • how many runtimes are there?
  • what configuration do they have?
• Tasks spawned within the context of another task take that task as a parent
  • This causes some issues because the parent task's span won't close until the child hasn't, which isn't what is actually happening in Tokio (issues in Console: tokio-rs/console#345, tokio-rs/console#412)
• Work stealing isn't visible (there is no concept of workers in the traces)

Describe the solution you'd like

I would like to add some additional traces to Tokio and modify others.

The proposed hierarchy is the following. Sub-items have their span's parent set. Values in parentheses are related fields which would be set. Note that all spans have a span ID implicitly, these are used for internal referencing.

• Runtime [0] (kind=current_thread, runtime.id=1, runtime.name="my runtime")
  • Worker [0] (name=my-worker-0)
  • Worker [1] …
  • Blocking thread [0]
  • Blocking thread [1] …
  • Task [0] (task.id=1)
    • Async Op [1] (resource.span.id=0, worker.span.id=1)
  • Task [1] (follows_from: Task [0], task.id=2)
  • Task [2] (task.id=3)
    • Async Op [5] (resource.id=0, worker.id=0)
• Runtime [1] …
• Resource [0] (creator.task.span.id=4, creator.runtime.span.id=0)
• Resource [1] ...

Further explanation of the structure:

1. Runtimes are top level (root) spans.
2. Workers belong to runtimes
3. Blocking threads belong to runtimes
   1. If a blocking thread is destroyed and another is created, it will have a different span (even if the id is the same)
4. Tasks belong to runtimes
   1. Tasks don’t have parent tasks, but they follow_from the task that spawned them (if any)
   2. This is a breaking change
5. Resources are top level (root) spans
   1. They may be shared between runtimes
   2. They reference the runtime (if any) and task (if any) that created the resource
6. Async ops belong to tasks (the task that executes the async op)
   1. They reference the resource that the async op belongs to
   2. They reference the worker id where they are executed

Ideally all these changes, or at least all the breaking ones would make it into the same Tokio release. Preferably with the addition of spans for Runtimes. This would allow Tokio Console to determine the version tokio tracing variant being used and read the traces accordingly.

Describe alternatives you've considered

Parts of this hierarchy can be implemented without the rest. But it makes sense to discuss the entirety.

Additional context

This is mostly considering being consumed by Tokio Console. However it would be interesting to hear from anyone who is using tokio traces in other ways.

[Darksonn]

This sounds reasonable to me. You should probably coordinate with @carllerche's scheduler refactor, which may change the same parts of the codebase.

[qRoC]

A task may outlive the context in which is was spawned (and frequently does).

@hds What about block_on?

This runs the given future on the current thread, blocking until it is complete

Looks like him span should have parent.

[hds]

A task may outlive the context in which is was spawned (and frequently does).

@hds What about block_on?

This runs the given future on the current thread, blocking until it is complete

Looks like him span should have parent.

This is something we should consider. We've taken it out for now, but a subscriber could still get the context for that span and store it if it needed to.