Other timing tools

[aminya]

Is there any other timing measure that shows the effect of using precompiles (in a dynamic manner without sysimage) other than what we have in @snoopi and snoopi_bot?

Is @timev a good measure?

[aminya]

I used @timev, and clearly shows the improvements 🚀 . People thought that the only improvement that we get is in the inference time.

For example, for Zygote:

Running @timev in running the tests shows that we get:

• 10s raw improvement
• 23 MB less memory usage
• 1GiB less memory allocation.

Details

```jl │ Precompile Deactivated Benchmark # for using Zygote: └ Inference time (ms): 23654.842 @timev result (This has some noise). 391.374697 seconds (628.72 M allocations: 31.522 GiB, 4.16% gc time) elapsed time (ns): 391374697041 gc time (ns): 16295209095 bytes allocated: 33846029551 pool allocs: 628513544 non-pool GC allocs:190530 malloc() calls: 11983 realloc() calls: 85 GC pauses: 503 full collections: 9 ``` ```jl │ Precompile Activated Benchmark ┌ Info: └ Inference time (ms): 22565.431 @timev result (This has some noise). 382.785964 seconds (605.11 M allocations: 30.360 GiB, 3.93% gc time) elapsed time (ns): 382785964109 gc time (ns): 15042733929 bytes allocated: 32598547445 pool allocs: 604912736 non-pool GC allocs:184276 malloc() calls: 11737 realloc() calls: 81 GC pauses: 467 full collections: 6 ```

https://github.com/aminya/Zygote.jl/runs/646094164?check_suite_focus=true#step:6:123

[aminya]

If this is the case, should I just use @timev for benchmarking isntead of @snoopi? People seem to have better intuition for it.

    julia_code_timev = """
    using $package_name
    @timev begin
        $(string(snoop_script));
    end
    @info("The above is @timev result (This has some noise).")
    """
    julia_cmd_timev = `julia --project=@. -e $julia_code_timev`

https://github.com/aminya/SnoopCompile.jl/blob/4c5556de806fd2c9fcd7c82e606c6162f102b195/src/bot/snoopi_bench.jl#L33-L40

[timholy]

@snoopi is for discovery and for measuring just one thing, inference time. @timev measures a whole bunch of things, including load time, LLVM time, execution time, etc. In the end it's closer to what people will want to reduce, so it's not bad to use it to decide whether the precompiles are worth it. E.g., #74.

In general I worry that too many people use SnoopCompile unthinkingly; for me, the "discovery" part is by far the most important and I generally prefer to write the precompile file by hand. But I recognize I'm probably an outlier. Recommending @timev might help people realize that, for example, setting a higher tmin (and thus precompiling less) might actually reduce times they care about.

[aminya]

People are less concerned with the precompiling time, but with its effect on the performance. Using @timev helps with this for sure. I think I can close this, as this will be solved in #71