Doxygen-like automatic generation of links to documentation from source code ?

[j-fu]

Hi, my be this is a little related to #1971 but I think it deserves it's own issue. Is there a chance to have links to documentation in the example code blocks ?

Doxygen can do this, see e.g. here.

I agree this is not easy, IMHO it would make much sense though.

[fredrikekre]

I was thinking about this a couple of weeks ago too. I think the tricky thing would be to figure out where to link. If there is only one docstring for a function that is easy enough, but it might be tricky to figure out which specific method to link to.

[j-fu]

Could JuliaSyntax.jl be helpful here ? (just guessing...)

[fredrikekre]

You would need to evaluate the code in order to get the types and then find the correct method so I don't think so. For e.g. @example blocks Documenter does run the code though, so potentially it could be possible to record the methods that are called somehow. For regular julia code blocks, perhaps one could link to the searchresult or something.

[fredrikekre]

Someone pointed out that https://github.com/lorenzoh/Pollen.jl supports this, so perhaps we can look at and/or borrow that implementation.

[j-fu]

I had a short look - he seems to use JuliaSyntax and my (quite coarse) impression is that he uses docstrings for functions and provides a list of methods under the function, see e.g. his xtree.jl - lool for cata or catafold. BTW once we discuss this I tag @lorenzoh ...

So what about just linking to the function docs as default ? If this is explainend well, docstrings could be arranged accordingly.

An addition to that which does not require to run the code could be some markup to overwrite these default links in the source examples like

foo(3)    # doclink foo(::Integer)
foo(3.0)  # doclink foo(::AbstractFloat)

which is then hidden in the output (a bit like the # hide stuff)

[staticfloat]

I think for an initial implementation merely jumping to a page that contains all methods for a particular function would be sufficient, but if you wanted to narrow down the dispatch a bit, it should be possible to use https://github.com/julia-vscode/StaticLint.jl to at least parse the source and determine some kind of bound for the types flowing into each function call. That should be fairly fast as well, as VSCode does this live as you're editing a file.

[staticfloat]

Pinging @pfitzseb to see what he thinks of this plan. I think this could be a big usability improvement for packages with complex examples.

[pfitzseb]

Seems reasonable to me, except for the fact that I'd be somewhat hesitant building infrastructure on top of StaticLint given that it's notoriously hard to work with.

[fredrikekre]

I wonder if https://github.com/JuliaDocs/DocInventories.jl and https://github.com/JuliaDocs/DocumenterInterLinks.jl could be useful for an initial implementation here (cc @goerz). Could even be implemented as a final pass on the docs after they are rendered.

[goerz]

Yes, the inventories would probably be very helpful in figuring out where to link to. This still seems like a non-trivial feature to implement.

[staticfloat]

I ended up hacking together something very simple using JuliaSyntax:

using JuliaSyntax: ParseStream, parse!, build_tree, children, SyntaxNode, @K_str, is_infix_op_call, child_position_span

function find_function_calls(tree, function_list::Vector)
    if kind(tree) == K"call" || kind(tree) == K"dotcall"
        push!(function_list, tree)
    end
    for node in children(tree)
        find_function_calls(node, function_list)
    end
end

function interesting_functions(node)
    # Ignore juxtaposition, which for some reason is parsed as a call
    if kind(node.children[1]) == K"juxtapose"
        return false
    end

    # Ignore infix operators in general
    if is_infix_op_call(head(node))
        return false
    end
    return true
end

# Heuristic to return a string representation of the given node,
# assuming its the target of a `call`.
function function_call_name(node)
    if kind(node) == K"."
        return string(
            function_call_name(node.children[1]),
            ".",
            function_call_name(node.children[2]),
        )
    elseif kind(node) == K"quote"
        return function_call_name(node.children[1])
    else
        return string(node)
    end
end

function function_call_data(node)
    name, pos, span = child_position_span(node.children[1])
    return function_call_name(name), pos, span
end

recursive_isdefined(mod::Module, name::Symbol) = isdefined(mod, name)
recursive_isdefined(mod::Module, name::QuoteNode) = recursive_isdefined(mod, name.value)
function recursive_isdefined(mod::Module, name::Expr)
    return name.head == :(.) &&
           recursive_isdefined(mod, name.args[1]) &&
           recursive_isdefined(mod, names.args[2])
end

# Find all function calls in `code`, and if they are in `mod`
# insert a reference to the docs of that function.
function annotate_function_calls(code::String, mod::Module)
    st = ParseStream(code)
    parse!(st)
    tree = build_tree(SyntaxNode, st)
    Main.tree = tree
    function_list = []
    find_function_calls(tree, function_list)
    filter!(interesting_functions, function_list)

    # For each function call, get its name and location in the string
    # then replace it with `[function_name](@ref)`:
    last_code_pos = firstindex(code)
    output = IOBuffer()
    for f in function_list
        name, pos, span = function_call_data(f)
        if recursive_isdefined(mod, Meta.parse(name))
            print(output, code[last_code_pos:pos-1])
            print(output, "[$(name)](@ref)")
            last_code_pos = pos + span
        end
    end
    print(output, code[last_code_pos:end])

    return String(take!(output))
end

If you run it like this:

module TestMod
foo(x) = x
end

code = """
foo(1)
bar(2)

[foo(x) for x in 1:10]

function do_work(xs)
    return foo.(xs) .+ 1
end
"""

println(annotate_function_calls(code, TestMod))

You get this:

[foo](@ref)(1)
bar(2)

[[foo](@ref)(x) for x in 1:10]

function do_work(xs)
    return [foo](@ref).(xs) .+ 1
end

It's pretty easy to then pre-process some .md files to have these [function_name](@ref) strings inserted into them in the right places, but unfortunately Documenter doesn't parse these ref links inside of code blocks. Is there a way to get code block styling, but still parsing the ref-links?

[fredrikekre]

Might be easier to insert the links directly then using something like the DocInventory linked above. This will most likely also trip up the Julia syntax highlighting which is why I suggested to post-process the rendered html (with prerendering of the syntax hightligting instead of doing it "live" with javascript).