Open timholy opened 3 months ago
Learning materials:
More specific code examples:
Method
s rather than Module
s for the output, but otherwise use the same visit
logic.Thanks Tim, this is really helpful! I will give it a try.
Is there a way to get lowered top-level code in a module? I'm having trouble finding a way to do that, which would be nice since then we might be able to drop parsing altogether. (Though already using lowered code for methods would be a win).
You mean stuff that runs at module-definition time but doesn't result in a method signature? Great question, and AFAIK the answer is no. For that you do have to parse 😢.
You could probably call Meta.lower
on it, though, if you want to use Julia's scoping to resolve the references.
Having some issues improving accuracy using lowered code:
unzip
statically because it is not a leaf node of the syntax tree (instead the cmdstr is the leaf). In some cases (see next bullet) I can detect unzip
but I can't necessarily tell if it was qualified or not. For the sake of finding a non-stale import it's OK, but I can't detect an unqualified usage of an implicitly imported name int his circumstance.module Exporter123
export exported_a
exported_a() = "hi"
end # Exporter123
module TestInterpolation
using ..Exporter123
function register_steelProfile()
function file()
return print(`$(exported_a())`)
end
end
end # TestInterpolation
and use analyze_locals_nonrecursive
from my branch https://github.com/ericphanson/ExplicitImports.jl/blob/da63656eb790a8045adaac7da72367b52b27bd3c/src/analyze_lowered.jl#L38-L101, I get:
julia> lookup_lowered = analyze_locals_nonrecursive(TestInterpolation)
Dict{@NamedTuple{file::String, line::Int64, name::Symbol}, Module} with 7 entries:
(file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 11, name = :cmd_gen) => Base
(file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 6, name = :_call_latest) => Core
(file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 11, name = :tuple) => Core
(file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 6, name = :TestInterpolation) => Main.TestInterpolation
(file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 10, name = Symbol("#file#1")) => Main.TestInterpolation
(file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 6, name = :include) => Base
(file = "/Users/eph/ExplicitImports/test/test_interpolation.jl", line = 6, name = :eval) => Core
Note: no Exporter123
. Then if I do
julia> TestInterpolation.register_steelProfile()()
`hi`
julia> lookup_lowered = analyze_locals_nonrecursive(TestInterpolation)
Dict{@NamedTuple{file::String, line::Int64, name::Symbol}, Module} with 9 entries:
(file = "/Users/eph/ExplicitImports/test/test_interpo… => Main.TestInterpolation
(file = "/Users/eph/ExplicitImports/test/test_interpo… => Base
(file = "/Users/eph/ExplicitImports/test/test_interpo… => Base
(file = "/Users/eph/ExplicitImports/test/test_interpo… => Core
(file = "/Users/eph/ExplicitImports/test/test_interpo… => Main.Exporter123
(file = "/Users/eph/ExplicitImports/test/test_interpo… => Main.TestInterpolation
(file = "/Users/eph/ExplicitImports/test/test_interpo… => Core
(file = "/Users/eph/ExplicitImports/test/test_interpo… => Base
(file = "/Users/eph/ExplicitImports/test/test_interpo… => Core
I find it. Note the same thing happens without the nested function, I think it's from the command interpolation.
So I think I can improve accuracy of my scope resolution during parsing by finding globalrefs on the same line corresponding to the same name and checking their owning module, but I can't replace parsing nor add coverage to areas I don't handle well during parsing (e.g. macros, interpolation, etc).
However, I do think we can solve https://github.com/ericphanson/ExplicitImports.jl/issues/73 thanks to lowered code info, at least in the case where the name in question (Transpose
, being exported by LinearAlgebra and a global in the module due to @enumx
) is being used in a function (so I can get the lowered info). This is a case where parsing "works", but my scope resolution is inaccurate, since it didn't recognize @enumx
as declaring new globals in the module. For usages of Transpose
where I can get at the lowered code, I can check the owning module (matching the parsed info against the file/line/name from the lowered code) and realize it is local to our module rather than an external name.
This isn't enough to actually close the issue, since the line @enumx ApplyStrategy Transpose Inplace
will still trigger a FP where we think Transpose
is coming from LinearAlgebra (since we don't have lowered code from the toplevel stuff). But we would be able to identify usages in functions.
I can't determine if a name is qualified or not from the global ref / lowered code information
Oof. I see why this matters, and I didn't envision that when I opened this issue.
This should become easy once JuliaLowering lands (which won't be before 1.13, and of course there are no guarantees it will happen then). I would personally be tempted to wait for that to happen before pursuing this further, but that's a personal choice.
I think it's from the command interpolation
That seems very likely. I think your visit
f
function should exit with return !isa(item, Method)
. Here's the issue: if m
is a Method
, then m.specializations
holds all compiled MethodInstances
. If you descend into them, then what you find will depend on what has been run. If you don't want that to be true, then just truncate the visitation pattern once you reach a Method
.
If ExplicitImports were to iterate over all the names in the package, and then for every item that is a method, ask for
Base.uncompressed_ast
, then you could tally all theGlobalRef
s that appear in the lowered code. This would presumably make your scope resolution fully accurate.JuliaInterpreter does lots of shenanigans with lowered code, including replacing all
GlobalRefs
with the actual object (to save the cost of scope-resolution at runtime). So its code might be a good reference if you need pointers.