Add `report_invalidations` to tabulate invalidations summary

[charleskawczynski]

Hello 👋🏻! There has been a lot of activity around hunting down invalidations lately, for example:

• julia-invalidations
• discourse thread
• SciML issue

As mentioned in this comment of the discourse thread, I've added a function called report_invalidations to ReportMetrics.jl which summarizes the output of SnoopCompileCore.@snoopr with quantitative and actionable information.

I'd like to call this function from the julia-invalidations github action. I thought it might be cleaner to add this functionality to SnoopCompile directly (rather than a separate package) since it's relatively light weight (barring PrettyTables) and uses SnoopCompile internals.

The ReportMetrics.jl version of the printed table looks like this:

[ Info: Number of invalidations for SomeWork: 7
[ Info: SomeWork: 6 total method invalidations
┌───────────────────────────────────┬─────────────┬───────────────┬─────────────────┐
│ <file name>:<line number>         │ Method Name │ Invalidations │ Invalidations % │
│                                   │             │    Number     │     (xᵢ/∑x)     │
├───────────────────────────────────┼─────────────┼───────────────┼─────────────────┤
│ SnoopCompile.jlirrationals.jl:182 │  BigFloat   │       1       │       50        │
│ SnoopCompile.jlirrationals.jl:182 │  BigFloat   │       1       │       50        │
│ SnoopCompile.jlirrationals.jl:192 │   Float32   │       0       │        0        │
│ SnoopCompile.jlirrationals.jl:191 │   Float64   │       0       │        0        │
│ SnoopCompile.jlirrationals.jl:192 │   Float32   │       0       │        0        │
│ SnoopCompile.jlirrationals.jl:191 │   Float64   │       0       │        0        │
└───────────────────────────────────┴─────────────┴───────────────┴─────────────────┘

[codecov[bot]]

Codecov Report

Base: 84.08% // Head: 83.70% // Decreases project coverage by -0.38% :warning:

Coverage data is based on head (5b4a42f) compared to base (dcae62d). Patch coverage: 87.50% of modified lines in pull request are covered.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #303 +/- ## ========================================== - Coverage 84.08% 83.70% -0.39% ========================================== Files 17 18 +1 Lines 2162 2221 +59 ========================================== + Hits 1818 1859 +41 - Misses 344 362 +18 ``` | [Impacted Files](https://codecov.io/gh/timholy/SnoopCompile.jl/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Tim+Holy) | Coverage Δ | | |---|---|---| | [src/invalidations.jl](https://codecov.io/gh/timholy/SnoopCompile.jl/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Tim+Holy#diff-c3JjL2ludmFsaWRhdGlvbnMuamw=) | `74.06% <85.71%> (-3.74%)` | :arrow_down: | | [src/report\_invalidations.jl](https://codecov.io/gh/timholy/SnoopCompile.jl/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Tim+Holy#diff-c3JjL3JlcG9ydF9pbnZhbGlkYXRpb25zLmps) | `86.66% <86.66%> (ø)` | | | [src/SnoopCompile.jl](https://codecov.io/gh/timholy/SnoopCompile.jl/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Tim+Holy#diff-c3JjL1Nub29wQ29tcGlsZS5qbA==) | `100.00% <100.00%> (ø)` | | | [src/parcel\_snoopi\_deep.jl](https://codecov.io/gh/timholy/SnoopCompile.jl/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Tim+Holy#diff-c3JjL3BhcmNlbF9zbm9vcGlfZGVlcC5qbA==) | `90.01% <0.00%> (+0.10%)` | :arrow_up: | | [src/parcel\_snoopi.jl](https://codecov.io/gh/timholy/SnoopCompile.jl/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Tim+Holy#diff-c3JjL3BhcmNlbF9zbm9vcGkuamw=) | `95.00% <0.00%> (+0.85%)` | :arrow_up: | | [SnoopPrecompile/src/SnoopPrecompile.jl](https://codecov.io/gh/timholy/SnoopCompile.jl/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Tim+Holy#diff-U25vb3BQcmVjb21waWxlL3NyYy9Tbm9vcFByZWNvbXBpbGUuamw=) | `94.11% <0.00%> (+6.61%)` | :arrow_up: | | [SnoopCompileCore/src/snoopi\_deep.jl](https://codecov.io/gh/timholy/SnoopCompile.jl/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Tim+Holy#diff-U25vb3BDb21waWxlQ29yZS9zcmMvc25vb3BpX2RlZXAuamw=) | `100.00% <0.00%> (+8.00%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Tim+Holy). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Tim+Holy)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[charleskawczynski]

bump 🙂

[charleskawczynski]

Bump! 🙂

[timholy]

I think we could have this, but I'd rather it be loaded under a @require PrettyTables = ... block. Often the best practice is to define the function

"""
Lotsa docs

Using `tabulated_invalidations` requires that you first load the PrettyTables package.
"""
function tabulated_invalidations end      # this defines a function with no methods, but the docstring is available.

as a "stub" in the main package, and then in __init__()

function __init__()
    # other stuff
    @require PettyTables = sha include("tabulated.jl")
end

and put the tabulated_invalidations methods in "tabulated.jl".

[charleskawczynski]

Ok, this now works I think as expected, here's a mwe (assuming some packages are loadable):

import SnoopCompileCore
invalidations = SnoopCompileCore.@snoopr begin
   using ForwardDiff
end;
import SnoopCompile
using PrettyTables # needed for report_invalidations to be defined
SnoopCompile.report_invalidations(;
   invalidations,
   process_filename = x -> last(split(x, ".julia/packages/"))
)

Which produces

┌ Warning: Could not attribute the following delayed invalidations:
└ @ SnoopCompile ~/Dropbox/Caltech/work/dev/clones/forks/SnoopCompile.jl/src/invalidations.jl:552
Any[MethodInstance for eltype(::Type{A} where A<:(NamedTuple{(:init,), _A} where _A<:Tuple{AbstractDict})), MethodInstance for eltype(::Type{T} where T<:Tuple{AbstractDict}), MethodInstance for eltype(::Type{A} where A<:(NamedTuple{(:maxlog, :form), _A} where _A<:Tuple{Int64, Type})), MethodInstance for eltype(::Type{T} where T<:Tuple{Int64, Type})] invalidated Core.MethodInstance[MethodInstance for StaticArrays.var"#s25#170"(::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any), MethodInstance for StaticArrays._precompile_()]
Any[MethodInstance for eltype(::Type{T} where T<:Tuple{AbstractDict}), MethodInstance for eltype(::Type{A} where A<:(NamedTuple{(:init,), _A} where _A<:Tuple{AbstractDict}))] invalidated MethodInstance for MacroTools.allbindings(::Any, ::Vector{Any})
[ Info: 233 methods invalidated for 6 functions
┌─────────────────────────────────────────────┬───────────────┬───────────────┬─────────────────┐
│ <file name>:<line number>                   │ Function Name │ Invalidations │ Invalidations % │
│                                             │               │               │     (xᵢ/∑x)     │
├─────────────────────────────────────────────┼───────────────┼───────────────┼─────────────────┤
│ ForwardDiff/QdStj/src/dual.jl:427           │ promote_rule  │      209      │       82        │
│ StaticArrays/x7lS0/src/abstractarray.jl:150 │    similar    │      18       │        7        │
│ StaticArrays/x7lS0/src/SizedArray.jl:88     │    convert    │      15       │        6        │
│ StaticArrays/x7lS0/src/mapreduce.jl:253     │   foldl##kw   │       8       │        3        │
│ ForwardDiff/QdStj/src/dual.jl:145           │    isequal    │       3       │        1        │
│ StaticArrays/x7lS0/src/SOneTo.jl:57         │  getproperty  │       3       │        1        │
└─────────────────────────────────────────────┴───────────────┴───────────────┴─────────────────┘

[charleskawczynski]

Based on https://github.com/JuliaDiff/ForwardDiff.jl/issues/620#issuecomment-1370119947 and https://github.com/JuliaDiff/ForwardDiff.jl/issues/620#issuecomment-1370142934, should we update how report_invalidations works? cc @ChrisRackauckas @timholy

[charleskawczynski]

I think one last thing that we could do, that would really put this over the top in terms of ease of use, would be if we followed through with https://github.com/CliMA/ReportMetrics.jl/issues/7#issuecomment-1015618744, and made the file / line number hypertext links through PrettyTables (which I think is supported here).

I think this can be a followup feature, though. Thumbs up to open an issue 🙂

[timholy]

Thanks! This is a nice addition. Release of SnoopCompile 2.10.0 is underway.