[rfc] Support per-process catapult trace information

[ejgallego]

As discussed in the meeting, it would be great to allow --trace-file to collect timing information provided by sub-processes, for example, OCaml could provide information about time spent in each phase or Coq could provide information about time spent in proofs.

I think there are two design choices to make:

• how to have the sub-processes communicate the timing information back to Dune. It seems to me that the easiest way is to make Stats.with_process aware of the trace output file, and include that information once the process has finished. This requires some extending of Fiber.t tho.
• how to instrument the corresponding call to processes when --with-trace is provided. As of today this info doesn't really live in the context so we'll likely have to propagate it there.

Thoughts?

Related issues:

• https://github.com/coq/coq/issues/11542
• https://github.com/ocaml/ocaml/pull/9082

[rgrinberg]

Just discussed this with Andrey and Pierre, and there's a definitely some demand for this feature. Here's a simple design that might work:

Whenever --trace-file is set:

1. dune will create a temporary directory for every action
2. It will set the path toDUNE_TRACE_FILE to this directory
3. Once the action executes, dune will collect all the files in that directory and them to the trace.

@snowleopard how does that sound?

[snowleopard]

@rgrinberg Sounds good but I don't follow this bit:

1. dune will create a temporary directory for every action

We don't need a separate directory for every action. A single directory (say, _build/.trace) should be sufficient, right? Then, for every action we set DUNE_TRACE_FILE=_build/.trace/<action_digest> so that the action can write to it.

[rgrinberg]

We don't need a separate directory for every action. A single directory (say, _build/.trace) should be sufficient, right? Then, for every action we set DUNE_TRACE_FILE=_build/.trace/ so that the action can write to it.

In my scheme, I allow for subcomponents of a particular action to write their own metrics as well. For example, if we run a ppx action, and different passes of the rewriter want to emit their own metrics - they're allowed to do so without worrying about the file being written to concurrently from somewhere else.

[snowleopard]

I see, that makes sense. What I'd like to avoid is the need to eagerly create that directory for every action, even though most of them don't need it (at least at first). Do you think we can require the actions that want to produce custom traces to mkdir that directory?

[rgrinberg]

I'm personally not worried about it as we create a ton of directories in dune builds and they never show up as overhead. Though allowing the actions to create them sounds fine as well to me.