Remove performance instrumentation?

[dhil]

The compilation unit performance.ml implements a means for intrinsic measurements of Links. It produces various runtime statistics of various phases in the Links compiler. It seems to be largely unused, yet it is fairly pervasive throughout the compiler. Maybe we can get rid of it?

We may want to more extensively benchmark Links in which case it may makes sense to keep performance.ml, however, the current iteration of it is somewhat ad-hoc. It would be preferable if the output were structured and easily translatable. Maybe we can use a "standard" extrinsic measurement tool like gperf to generate the statistics of interest.

Discuss.

[frank-emrich]

All I can find about gperf relates to generating hash functions. Did you mean perf?

I've used perf before to benchmark Links, and I'm not sure if using it would easily give you the same results as what performance.ml provides. The latter gives you high-level information (how long did type-checking take?), whereas perf gives you details (how often did we call function foo and how long took its executions in total?)

Currently, performance is only used in Loader, Backend, Driver, and Links. I agree that the way we currently use it is annoying, because of the laziness. Maybe some good old imperative code would make things less painful? Instead of

(** optimise and evaluate a program *)
let process_program
      (interacting : bool)
      (envs : evaluation_env)
      (program : Ir.program)
      external_files
          : (Value.env * Value.t) =
  let (valenv, nenv, tyenv) = envs in
  let tenv = (Var.varify_env (nenv, tyenv.Types.var_env)) in

  let perform_optimisations = Settings.get Backend.optimise && not interacting in

  let (globals, _main) as post_backend_pipeline_program =
    Backend.transform_program perform_optimisations tenv program in

  (if Settings.get typecheck_only then exit 0);

  Webserver.init (valenv, nenv, tyenv) globals external_files;

  lazy (Eval.run_program valenv post_backend_pipeline_program) |>measure_as<| "run_program"

let process_program  interacting envs program external_files =
  lazy (process_program  interacting envs program external_files) |>measure_as<| "process_program"

how about

(** optimise and evaluate a program *)
let process_program
      (interacting : bool)
      (envs : evaluation_env)
      (program : Ir.program)
      external_files
          : (Value.env * Value.t) =
  Performance.start_benchmark "process_program";
  let (valenv, nenv, tyenv) = envs in
  let tenv = (Var.varify_env (nenv, tyenv.Types.var_env)) in

  let perform_optimisations = Settings.get Backend.optimise && not interacting in

  let (globals, _main) as post_backend_pipeline_program =
    Backend.transform_program perform_optimisations tenv program in

  (if Settings.get typecheck_only then exit 0);

  Webserver.init (valenv, nenv, tyenv) globals external_files;

  Performance.start_benchmark "run_program";
  let result = (Eval.run_program valenv post_backend_pipeline_program) in
  Performance.stop_benchmark "run_program";
  Performance.stop_benchmark "process_program";
  result

This is more brittle in the sense that you have to start and stop the benchmarking yourself, but given the fact that we only have a handful of things that we want to benchmark, that may actually simplify things.

[jamescheney]

I meant gprof, which used to be possible to use with ocamlopt but no longer.

https://caml.inria.fr/pub/docs/manual-ocaml/profil.html

[jstolarek]

It would be nice to have a way of measuring compiler performance in a reliable way. Indeed, performance.ml seems a bit ad-hoc but, especially in the way results are reported, but it also looks like a good starting point for further improvements. I don't have a strong opinion whether it should go or stay. It doesn't feel like removing it will really make the code cleaner - it does not seem to be very pervasive - but at the same time I think no one intends to work on improving this at the moment. Maybe this would be a good topic for an intern summer project?

[jamescheney]

Is profiling an application of algebraic effects/handlers?

[dhil]

Is profiling an application of algebraic effects/handlers?

I believe yes. We can phrase the current implementation in terms of operations and effect handlers in the following way: Suppose we have an effect signature of two operations {Start:1 -> Int, Stop:int -> 1}. Then the measure function could be implemented as

let measure : (1 -> 'a) -> 'a
  = fun f -> 
     let t0 = perform Start () (* Performs the start operation. *)
     let x = f () in
     perform Stop t0; (* Performs the stop operation. *)
     x (* returns the value *)

We would run an application of measure under different handler depending on whether profiling is enabled. Supposing profiling is disabled. then we can simply interpret Start and Stop by ignoring them, e.g.

match measure f with
| Start _, k -> continue k 0
| Stop _, k -> continue k ()
| v -> v

When profiling is enabled, we interpret the operation by starting and stopping the watch

match measure f with
| Start _, k -> continue k (Unix.time ())
| Stop t0, k -> 
   printf "Time elapsed %d\n" (Unix.time () - t0);
   continue k ()
| v -> v

Using a parameterised handler we could even internalise the watch state within the handler, thus simplying the signature: {Start: 1 -> 1, Stop : 1 -> 1}.

[dhil]

It seems like the consensus is to keep the performance instrumentation. Thus I am closing the issue for now.