Add support for Cthulhu

[timholy]

This is the final tool (so far) for fixing invalidations. ascend makes it much easier to identify a fruitful place to intervene and fix inference problems.

This also adds tools for bumping the versions across-the-board. What do you think of this, @aminya? Overall I think our best strategy is to keep all version numbers in lock-step, and that's the workflow made easy with these tools. When you want to bump the version, you say bump_version(v"1.7.0"). After the PR merges, update your local master branch to the latest, check out a branch of registries/General, and then register_all(). Then submit your registries/General as a PR.

[timholy]

Argh, we're being bitten by https://github.com/JuliaLang/Pkg.jl/issues/1874. Naturally. I'd forgotten about that for a moment.

Have to switch to other things, will consider a fix soon.

[aminya]

Thanks for this! I think this bump_version should eventually go to Pkg. I will take a more thorough look soon.

[KristofferC]

Thanks for this! I think this bump_version should eventually go to Pkg.

Probably better in RegistryTools or something similar. Remember that Pkg has the disadvantage of being a stdlib and is thus slow to make fixes for and should not ideally break it's API etc. Seems much better for something like this somewhere without these constraints.

[timholy]

Thanks @KristofferC for suggesting the manifest.

@aminya, I switched the doc-build to nightly, since only nightly has the capabilities for the invalidations. I also fixed a couple of issues with the bot docs that emerged due to the split. Unfortunately we have two broken hyperlinks, from @snoopr to invalidation_trees and back, due to the fact that neither module knows about the other. I don't see a good fix for that. It's only an issue in the reference section.

[aminya]

Locally the tests fail for me.

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0-DEV.290 (2020-06-23)
 _/ |\__'_|_|_|\__'_|  |  Commit b59417235f (0 days old master)
|__/                   |

(@v1.6) pkg> activate .
 Activating environment at `C:\Users\yahyaaba\Documents\GitHub\Julia\SnoopCompile.jl\Project.toml`

(SnoopCompile) pkg> instantiate

julia> Pkg.test()
    Testing SnoopCompile
┌ Error: Pkg.Resolve.ResolverError("Unsatisfiable requirements detected for package Cthulhu [f68482b8]:\n Cthulhu [f68482b8] log:\n ├─possible versions are: [0.1.0-0.1.2, 1.0.0-1.0.2, 1.1.0-1.1.2] or uninstalled\n └─restricted to versions 1.2.0-1 by SnoopCompileAnalysis [9ea4277c] — no versions left\n   └─SnoopCompileAnalysis [9ea4277c] log:\n     ├─possible versions are: 1.7.0 or uninstalled\n     ├─restricted to versions 1.7.0-1 by SnoopCompileBot [1d5e0e55], leaving only versions 1.7.0\n     │ └─SnoopCompileBot 
[1d5e0e55] log:\n     │   ├─possible versions are: 1.7.0 or uninstalled\n     │   └─SnoopCompileBot [1d5e0e55] is fixed to version 1.7.0\n     └─SnoopCompileAnalysis [9ea4277c] is fixed to version 1.7.0", nothing)
└ @ Pkg.Operations D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:1410
ERROR: Unsatisfiable requirements detected for package Cthulhu [f68482b8]:
 Cthulhu [f68482b8] log:
 ├─possible versions are: [0.1.0-0.1.2, 1.0.0-1.0.2, 1.1.0-1.1.2] or uninstalled
 └─restricted to versions 1.2.0-1 by SnoopCompileAnalysis [9ea4277c] — no versions left
   └─SnoopCompileAnalysis [9ea4277c] log:
     ├─possible versions are: 1.7.0 or uninstalled
     ├─restricted to versions 1.7.0-1 by SnoopCompileBot [1d5e0e55], leaving only versions 1.7.0
     │ └─SnoopCompileBot [1d5e0e55] log:
     │   ├─possible versions are: 1.7.0 or uninstalled
     │   └─SnoopCompileBot [1d5e0e55] is fixed to version 1.7.0
     └─SnoopCompileAnalysis [9ea4277c] is fixed to version 1.7.0
Stacktrace:
 [1] check_constraints(::Pkg.Resolve.Graph) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Resolve\graphtype.jl:935
 [2] Pkg.Resolve.Graph(::Dict{Base.UUID,Set{VersionNumber}}, ::Dict{Base.UUID,Dict{VersionNumber,Dict{String,Base.UUID}}}, ::Dict{Base.UUID,Dict{VersionNumber,Dict{String,Pkg.Types.VersionSpec}}}, ::Dict{Base.UUID,String}, ::Dict{Base.UUID,Pkg.Types.VersionSpec}, ::Dict{Base.UUID,Pkg.Resolve.Fixed}, ::Bool) at 
D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Resolve\graphtype.jl:362     
 [3] deps_graph(::Pkg.Types.Context, ::Dict{Base.UUID,String}, ::Dict{Base.UUID,Pkg.Types.VersionSpec}, 
::Dict{Base.UUID,Pkg.Resolve.Fixed}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:493
 [4] resolve_versions!(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:383
 [5] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Pkg.Types.UpgradeLevel) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:1209
 [6] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, update_registry::Bool, kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:245
 [7] up(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:level, :mode, :update_registry, :io),Tuple{Pkg.Types.UpgradeLevel,Pkg.Types.PackageMode,Bool,Base.DevNull}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:68
 [8] resolve(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:251
 [9] (::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec})() at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:1412
 [10] with_temp_env(::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec}, ::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:1331
 [11] (::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String})(::String) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:1402
 [12] mktempdir(::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String}, ::String; prefix::String) at .\file.jl:682
 [13] mktempdir(::Function, ::String) at .\file.jl:680 (repeats 2 times)
 [14] sandbox(::Function, ::Pkg.Types.Context, ::Pkg.Types.PackageSpec, ::String, ::String, ::Pkg.Types.Project) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:1369
 [15] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Operations.jl:1533
 [16] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:327
 [17] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:314
 [18] #test#61 at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:67 [inlined]
 [19] test at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:67 [inlined]
 [20] test(; name::Nothing, uuid::Nothing, version::Nothing, url::Nothing, rev::Nothing, path::Nothing, 
mode::Pkg.Types.PackageMode, subdir::Nothing, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:80
 [21] test() at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\API.jl:72
 [22] top-level scope at none:1

Cthulthu 1.2.0 is not installed automatically. Is it registered? @timholy

(SnoopCompile) pkg> add Cthulhu
   Updating registry at `C:\Users\yahyaaba\.julia\registries\General`
┌ Warning: Some registries failed to update:
│     — `C:\Users\yahyaaba\.julia\registries\General` — registry dirty
└ @ Pkg.Types D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Types.jl:1194  
  Resolving package versions...
ERROR: Unsatisfiable requirements detected for package Cthulhu [f68482b8]:
 Cthulhu [f68482b8] log:
 ├─possible versions are: [0.1.0-0.1.2, 1.0.0-1.0.2, 1.1.0-1.1.2] or uninstalled
 └─restricted to versions 1.2.0-1 by SnoopCompileAnalysis [9ea4277c] — no versions left
   └─SnoopCompileAnalysis [9ea4277c] log:
     ├─possible versions are: 1.7.0 or uninstalled
     ├─restricted to versions 1.7.0-1 by SnoopCompileBot [1d5e0e55], leaving only versions 1.7.0        
     │ └─SnoopCompileBot [1d5e0e55] log:
     │   ├─possible versions are: 1.7.0 or uninstalled
     │   └─SnoopCompileBot [1d5e0e55] is fixed to version 1.7.0
     └─SnoopCompileAnalysis [9ea4277c] is fixed to version 1.7.0

[aminya]

Unfortunately we have two broken hyperlinks, from @snoopr to invalidation_trees and back, due to the fact that neither module knows about the other. I don't see a good fix for that. It's only an issue in the reference section.

This is this issue. I suggested a possible solution some time back, but I got no reaction. https://github.com/JuliaDocs/Documenter.jl/issues/688

We can use full links instead of [](@ref), to solve that temporarily

[timholy]

Cthulthu 1.2.0 is not installed automatically. Is it registered?

Yep, just update your packages. https://github.com/JuliaDebug/Cthulhu.jl/releases/tag/v1.2.0

[aminya]

Cthulthu 1.2.0 is not installed automatically. Is it registered?

Yep, just update your packages. JuliaDebug/Cthulhu.jl@v1.2.0 (release)

It might be an issue from my end. I fail to update my registry.

(@v1.6) pkg> up
   Updating registry at `C:\Users\yahyaaba\.julia\registries\General`
┌ Warning: Some registries failed to update:
│     — `C:\Users\yahyaaba\.julia\registries\General` — registry dirty
└ @ Pkg.Types D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Types.jl:1194
No Changes to `C:\Users\yahyaaba\.julia\environments\v1.6\Project.toml` 
No Changes to `C:\Users\yahyaaba\.julia\environments\v1.6\Manifest.toml`

[aminya]

OK, I can finally pass the tests locally.

It might be an issue from my end. I fail to update my registry.

(@v1.6) pkg> up
   Updating registry at `C:\Users\yahyaaba\.julia\registries\General`
┌ Warning: Some registries failed to update:
│     — `C:\Users\yahyaaba\.julia\registries\General` — registry dirty
└ @ Pkg.Types D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\Pkg\src\Types.jl:1194
No Changes to `C:\Users\yahyaaba\.julia\environments\v1.6\Project.toml` 
No Changes to `C:\Users\yahyaaba\.julia\environments\v1.6\Manifest.toml`

I had to delete my julia/registries to fix this.

[aminya]

I made a simple fix for cross-linking in Documenter. Let's see if it gets approved. https://github.com/JuliaDocs/Documenter.jl/pull/1351/files

[timholy]

This has now been rebased.

Long-term we definitely don't want the utils/register.jl, but in the short term I think let's live with it. The API is not yet general enough for RegistryTools.jl.

If you need to add your dev script then perhaps utils/ would be a better place.

[aminya]

Could we merge #103 and #109 to this PR as well?

[aminya]

I could not rebase, the conflicts were too much. I made a new PR to this branch: https://github.com/timholy/SnoopCompile.jl/pull/110

[timholy]

I'm going to hold off on this until we get 1.6.2 out which fixes the bot.

[timholy]

Given https://github.com/timholy/SnoopCompile.jl/issues/111#issuecomment-651245524 I think it means the bot is working again. Shall we merge this before or after fixing the conflicts in #110? Either is fine by me.

[aminya]

@timholy I have fixed the conflicts a couple of times so far. But they sneak in again.😉😄