cfunction issue with FunctionWrappers

[tkoolen]

From https://github.com/yuyichao/FunctionWrappers.jl/issues/8. This appears to be an issue in Base. Sorry for not having a smaller repro, I'm a bit pressed for time.

Consider (from https://github.com/yuyichao/FunctionWrappers.jl/issues/8#issuecomment-413189252), with FunctionWrappers 1.0.0:

module Parametron2

using FunctionWrappers: FunctionWrapper

struct LazyExpression{F, A}
    f::F
    args::A
    LazyExpression(f::F, args...) where {F} = new{F, typeof(args)}(f, args)
end

# Evaluation
@inline evalarg(x) = x
@inline evalarg(x::LazyExpression) = x()
@inline (expr::LazyExpression{F, A})() where {F, A} = evalexpr(expr.f, expr.args)
@generated function evalexpr(f::F, args::Tuple{Vararg{Any, N}}) where {F, N}
    # equivalent to f(map(evalarg, args)...), minus the inference issues
    argexprs = [:(evalarg(args[$i])) for i = 1 : N]
    quote
        Base.@_inline_meta
        f($(argexprs...))
    end
end

# Wrapping
const WrappedExpression{T} = LazyExpression{FunctionWrapper{T, Tuple{}}, Tuple{}}

Base.convert(::Type{WrappedExpression{T}}, expr::LazyExpression) where {T} =
    LazyExpression(FunctionWrapper{T, Tuple{}}(expr))

function wrap(expr::Parametron2.LazyExpression)
    T = typeof(expr())
    convert(Parametron2.WrappedExpression{T}, expr)
end

end

module M

import Main.Parametron2

f1 = [1]
converted = Parametron2.wrap(Parametron2.LazyExpression(convert, Vector{Float64}, f1))
converted.f()

end

A possible result of pasting this into the REPL:

ERROR: TypeError: in CallWrapper, in cfunction, expected Core.Compiler.VarState(Core.Compiler.Const(false, false), false), got Array{Float64,1}
Stacktrace:
 [1] macro expansion at /home/twan/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:111 [inlined]
 [2] do_ccall at /home/twan/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:102 [inlined]
 [3] (::FunctionWrappers.FunctionWrapper{Array{Float64,1},Tuple{}})() at /home/twan/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:118
 [4] top-level scope at none:0

julia> versioninfo()
Julia Version 1.0.0
Commit 5d4eaca0c9 (2018-08-08 20:58 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6950X CPU @ 3.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, broadwell)

cc: @yuyichao

[tkoolen]

This is still present on latest master, 1.1.0-DEV.143, 8555aa0aa3.

[tkoolen]

So the way I'm working around this right now for Parametron is by basically calling the old cfunction backend instead of @cfunction. See https://github.com/tkoolen/Parametron.jl/blob/45a4bdcdd1ca6202d679861bf9db9583d4bb9ae5/src/FunctionWrappersQuickFix.jl#L86.

@chkwon is doing the same thing in PATHSolver.jl, see https://github.com/chkwon/PATHSolver.jl/blob/master/src/FunctionWrappersQuickFix.jl.

Could you please not remove jl_function_ptr and any other relevant parts of the old cfunction backend until this is fixed? It would kill our packages.

[chkwon]

Please...

[rdeits]

I just tried repeatedly pasting the above code into the REPL in the latest Julia nightly and was not able to reproduce the issue.

julia> module M

       import Main.Parametron2

       f1 = [1]
       converted = Parametron2.wrap(Parametron2.LazyExpression(convert, Vector{Float64}, f1))
       converted.f()

       end
Main.M

julia> versioninfo()
Julia Version 1.3.0-DEV.92
Commit 73366aef59 (2019-04-23 14:33 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E3-1535M v6 @ 3.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

perhaps it's fixed?

[yuyichao]

I remember there was a inference/compiler PR related to llvmcall with a very similar error message.

[rdeits]

Hm, looks like there's still a problem. I modified Parametron.jl to switch back to using FunctionWrappers.jl, and while most of the tests pass, I still get the following error:

Model: unconstrained: Error During Test at /home/rdeits/.julia/dev/Parametron/test/model.jl:37
  Got exception outside of a @test
  TypeError: in cfunction, expected Core.Compiler.Const(getindex, false), got Parametron.Functions.QuadraticFunction{Float64}
  Stacktrace:
   [1] macro expansion at /home/rdeits/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:111 [inlined]
   [2] do_ccall at /home/rdeits/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:102 [inlined]
   [3] FunctionWrapper at /home/rdeits/.julia/packages/FunctionWrappers/mJrYu/src/FunctionWrappers.jl:118 [inlined]
   [4] macro expansion at /home/rdeits/.julia/dev/Parametron/src/lazyexpression.jl:59 [inlined]
   [5] evalexpr at /home/rdeits/.julia/dev/Parametron/src/lazyexpression.jl:56 [inlined]
   [6] LazyExpression at /home/rdeits/.julia/dev/Parametron/src/lazyexpression.jl:53 [inlined]
   [7] update! at /home/rdeits/.julia/dev/Parametron/src/moi_interop.jl:133 [inlined]
   [8] update!(::Parametron.Model{Float64,OSQP.MathOptInterfaceOSQP.Optimizer}) at /home/rdeits/.julia/dev/Parametron/src/model.jl:137
   [9] solve! at /home/rdeits/.julia/dev/Parametron/src/model.jl:156 [inlined]
   [10] #test_unconstrained#1(::Float64, ::typeof(Main.ParametronTest.ModelTests.test_unconstrained), ::Parametron.Model{Float64,OSQP.MathOptInterfaceOSQP.Optimizer}, ::Array{Parametron.Functions.Variable,1}, ::Parametron.Parameter{Array{Float64,2},getfield(Main.ParametronTest.ModelTests, Symbol("##3#7")){Random.MersenneTwister},true}, ::Parametron.Parameter{Array{Float64,1},getfield(Main.ParametronTest.ModelTests, Symbol("##4#8")){Random.MersenneTwister},true}, ::Parametron.Parameter{Float64,getfield(Main.ParametronTest.ModelTests, Symbol("##5#9")){Base.RefValue{Float64}},false}) at /home/rdeits/.julia/dev/Parametron/test/model.jl:27
   [11] test_unconstrained(::Parametron.Model{Float64,OSQP.MathOptInterfaceOSQP.Optimizer}, ::Array{Parametron.Functions.Variable,1}, ::Parametron.Parameter{Array{Float64,2},getfield(Main.ParametronTest.ModelTests, Symbol("##3#7")){Random.MersenneTwister},true}, ::Parametron.Parameter{Array{Float64,1},getfield(Main.ParametronTest.ModelTests, Symbol("##4#8")){Random.MersenneTwister},true}, ::Parametron.Parameter{Float64,getfield(Main.ParametronTest.ModelTests, Symbol("##5#9")){Base.RefValue{Float64}},false}) at /home/rdeits/.julia/dev/Parametron/test/model.jl:27
   [12] top-level scope at /home/rdeits/.julia/dev/Parametron/test/model.jl:71

using Julia 1.3.0-DEV.92 as above.

[moustachio-belvedere]

Apologies if this adds unhelpful noise but I started experiencing the same issue but only at Julia v1.3, and in slightly modified form on v1.4 rc1. Versions 1.0, 1.1, 1.2 all worked fine. See full issue with MWE here

https://github.com/yuyichao/FunctionWrappers.jl/issues/12

[moustachio-belvedere]

Our nightly builds, with source code making use of FunctionWrappers.jl unchanged since previously failing, started working again. Presumably from the recent merged PR #39821 .

[tkoolen]

I tried the latest nightly (Version 1.7.0-DEV.751, Commit e2f5f26aec) with the tk/1.5 branch of Parametron (the package from which this issue originated), and FunctionWrappers v1.1.2. Running tests now results in:

Assertion failed: (SRet), function LocalScan, file /Users/julia/buildbot/worker/package_macos64/build/src/llvm-late-gc-lowering.cpp, line 1454.

i.e. this line: https://github.com/JuliaLang/julia/blob/4f5044a86b92b008708096918c6309e4503e8437/src/llvm-late-gc-lowering.cpp#L1454.

[tkoolen]

Ah, that is https://github.com/JuliaLang/julia/issues/40187 / https://github.com/yuyichao/FunctionWrappers.jl/issues/23.