allow `analyze` to accept URLs

[ericphanson]

I reorganized dispatch a little bit so that everything lowers to analyze_path once you've got the code locally somewhere. I wasn't totally sure what the best approach was here, but I thought it was good if there was a way to avoid the parsing when you already know it's local.

Let me know what you think!

[ericphanson]

One fun thing you can do with URLs is

using GitHub, AnalyzeRegistry
auth = GitHub.authenticate(ENV["GITHUB_AUTH"])
me = whoami(; auth)
my_julia_packages = filter!(endswith(".jl"), [ r.html_url.uri for r in repos(me)[1] if !r.fork])
results = analyze.(my_julia_packages)

to get an analysis of all your Julia packages (registered or not).

[giordano]

Uhm, this seems to have broken

julia> analyze(find_packages("DataFrames", "Flux"))
ERROR: UndefVarError: name not defined
Stacktrace:
  [1] analyze_path!(dest::String, repo::String; subdir::String, auth::GitHub.OAuth2)
    @ AnalyzeRegistry ~/.julia/dev/AnalyzeRegistry/src/AnalyzeRegistry.jl:288
  [2] analyze!(root::String, pkg::AnalyzeRegistry.RegistryEntry; auth::GitHub.OAuth2)
    @ AnalyzeRegistry ~/.julia/dev/AnalyzeRegistry/src/AnalyzeRegistry.jl:266
  [3] macro expansion
    @ ~/.julia/dev/AnalyzeRegistry/src/AnalyzeRegistry.jl:305 [inlined]
  [4] ##reducing_function#259#42
    @ ~/.julia/packages/FLoops/AN2BM/src/reduce.jl:450 [inlined]
  [5] next
    @ ~/.julia/packages/Transducers/VwORa/src/combinators.jl:261 [inlined]
  [6] next
    @ ~/.julia/packages/Transducers/VwORa/src/core.jl:289 [inlined]
  [7] macro expansion
    @ ~/.julia/packages/Transducers/VwORa/src/core.jl:181 [inlined]
  [8] _foldl_array
    @ ~/.julia/packages/Transducers/VwORa/src/processes.jl:187 [inlined]
  [9] __foldl__
    @ ~/.julia/packages/Transducers/VwORa/src/processes.jl:182 [inlined]
 [10] foldl_nocomplete
    @ ~/.julia/packages/Transducers/VwORa/src/processes.jl:356 [inlined]
 [11] _reduce_basecase(rf::Transducers.BottomRF{Transducers.AdHocRF{Nothing, typeof(identity), AnalyzeRegistry.var"##reducing_function#259#42"{GitHub.OAuth2, String}, typeof(identity), AnalyzeRegistry.var"##combine_function#260#43"}}, init::Transducers.OnInit{AnalyzeRegistry.var"#41#44"}, reducible::Transducers.SizedReducible{SubArray{AnalyzeRegistry.RegistryEntry, 1, Vector{AnalyzeRegistry.RegistryEntry}, Tuple{UnitRange{Int64}}, true}, Int64})
    @ Transducers ~/.julia/packages/Transducers/VwORa/src/threading_utils.jl:56
 [12] _reduce(ctx::Transducers.NoopDACContext, rf::Transducers.BottomRF{Transducers.AdHocRF{Nothing, typeof(identity), AnalyzeRegistry.var"##reducing_function#259#42"{GitHub.OAuth2, String}, typeof(identity), AnalyzeRegistry.var"##combine_function#260#43"}}, init::Transducers.OnInit{AnalyzeRegistry.var"#41#44"}, reducible::Transducers.SizedReducible{SubArray{AnalyzeRegistry.RegistryEntry, 1, Vector{AnalyzeRegistry.RegistryEntry}, Tuple{UnitRange{Int64}}, true}, Int64})
    @ Transducers ~/.julia/packages/Transducers/VwORa/src/reduce.jl:170
 [13] _reduce(ctx::Transducers.NoopDACContext, rf::Transducers.BottomRF{Transducers.AdHocRF{Nothing, typeof(identity), AnalyzeRegistry.var"##reducing_function#259#42"{GitHub.OAuth2, String}, typeof(identity), AnalyzeRegistry.var"##combine_function#260#43"}}, init::Transducers.OnInit{AnalyzeRegistry.var"#41#44"}, reducible::Transducers.SizedReducible{Vector{AnalyzeRegistry.RegistryEntry}, Int64})
    @ Transducers ~/.julia/packages/Transducers/VwORa/src/reduce.jl:179
 [14] _transduce_assoc_nocomplete
    @ ~/.julia/packages/Transducers/VwORa/src/reduce.jl:159 [inlined]
 [15] transduce_assoc(xform::Transducers.IdentityTransducer, step::Transducers.AdHocRF{Nothing, typeof(identity), AnalyzeRegistry.var"##reducing_function#259#42"{GitHub.OAuth2, String}, typeof(identity), AnalyzeRegistry.var"##combine_function#260#43"}, init::Transducers.OnInit{AnalyzeRegistry.var"#41#44"}, coll0::Vector{AnalyzeRegistry.RegistryEntry}; simd::Val{false}, basesize::Nothing, stoppable::Nothing, nestlevel::Nothing)
    @ Transducers ~/.julia/packages/Transducers/VwORa/src/reduce.jl:128
 [16] transduce
    @ ~/.julia/packages/Transducers/VwORa/src/executors.jl:50 [inlined]
 [17] transduce
    @ ~/.julia/packages/Transducers/VwORa/src/executors.jl:62 [inlined]
 [18] _fold
    @ ~/.julia/packages/FLoops/AN2BM/src/reduce.jl:481 [inlined]
 [19] _fold
    @ ~/.julia/packages/FLoops/AN2BM/src/reduce.jl:479 [inlined]
 [20] macro expansion
    @ ~/.julia/packages/FLoops/AN2BM/src/reduce.jl:461 [inlined]
 [21] analyze!(root::String, packages::Vector{AnalyzeRegistry.RegistryEntry}; auth::GitHub.OAuth2)
    @ AnalyzeRegistry ~/.julia/dev/AnalyzeRegistry/src/AnalyzeRegistry.jl:304
 [22] #46
    @ ~/.julia/dev/AnalyzeRegistry/src/AnalyzeRegistry.jl:347 [inlined]
 [23] mktempdir(fn::AnalyzeRegistry.var"#46#47"{GitHub.OAuth2, Vector{AnalyzeRegistry.RegistryEntry}}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:729
 [24] mktempdir (repeats 2 times)
    @ ./file.jl:727 [inlined]
 [25] analyze(p::Vector{AnalyzeRegistry.RegistryEntry}; auth::GitHub.OAuth2)
    @ AnalyzeRegistry ~/.julia/dev/AnalyzeRegistry/src/AnalyzeRegistry.jl:346
 [26] analyze(p::Vector{AnalyzeRegistry.RegistryEntry})
    @ AnalyzeRegistry ~/.julia/dev/AnalyzeRegistry/src/AnalyzeRegistry.jl:346
 [27] top-level scope
    @ REPL[7]:1

Both name and uuid are undefined. We clearly don't test this function :disappointed:

[ericphanson]

Oh oops! I'll put up a fix

[ericphanson]

Huh, it works for me?

julia> analyze(find_packages("DataFrames", "Flux"))
2-element Vector{AnalyzeRegistry.Package}:
 Package DataFrames:
  * repo: https://github.com/JuliaData/DataFrames.jl.git
  * uuid: a93c6f00-e57d-5684-b7b6-d8193f3e46c0
  * is reachable: true
  * lines of Julia code in `src`: 16531
  * lines of Julia code in `test`: 16606
  * has license(s) in file: MIT
    * filename: LICENSE.md
    * OSI approved: true
  * has documentation: true
  * has tests: true
  * has continuous integration: true
    * GitHub Actions
 Package Flux:
  * repo: https://github.com/FluxML/Flux.jl.git
  * uuid: 587475ba-b771-5e3f-ad9e-33799f191a9c
  * is reachable: true
  * lines of Julia code in `src`: 5074
  * lines of Julia code in `test`: 2167
  * has license(s) in file: MIT
    * filename: LICENSE.md
    * OSI approved: true
  * has documentation: true
  * has tests: true
  * has continuous integration: true
    * GitHub Actions
    * Buildkite

[ericphanson]

oooh I see, the error path is hit when the clone fails, and that path has undefined name

[giordano]

Yes, I'm struggling with cloning today and going down a bottomless rabbit hole: https://github.com/JuliaPackaging/Yggdrasil/pull/2708, https://github.com/JuliaVersionControl/GitCommand.jl/pull/42, https://github.com/giordano/AnalyzeRegistry.jl/pull/39 :stuck_out_tongue:

[ericphanson]

ooph, thanks for that! Hm ok, I'm not totally sure how to test the failure path here... Any ideas?

[giordano]

Pass a repo argument pointing to a non-existing URL?

[ericphanson]

oh right, sounds good aha