Using Soss with packages

[ptiede]

Hi, First I want to say sorry for not being able to find a minimal working example.

I am running into a strange stack trace that seems to occur when I define a Soss model in a package and then try to construct it. For example, suppose I have a package that looks like

module X
using Soss
using MeasureTheory
m = @model x=x begin
    mu ~ Normal(0.0, 1.0)
    y ~ Normal(mu*x, 1.0)
end
end

Then when I run into an issue when I try to run the model in a repl where I import module X, e.g.

using X
using Soss

x = 1.0
y = 10.0
xform(m(x=1.0)|(y=y,))

I get a stack trace like

Stacktrace:
  [1] getindex
    @ ./array.jl:801 [inlined]
  [2] getindex(::Vector{Tuple{Symbol, Any}}, ::UInt64)
    @ Base ./abstractarray.jl:1173
  [3] _lookup!(pool::Any, idx::Integer)
    @ GeneralizedGenerated.NGG ~/.julia/packages/GeneralizedGenerated/PV9u7/src/ngg/compact.jl:93
  [4] decompress_impl(encoded::UInt64, meta::Tuple)
    @ GeneralizedGenerated.NGG ~/.julia/packages/GeneralizedGenerated/PV9u7/src/ngg/compact.jl:166
  [5] decompress(encoded::Tuple)
    @ GeneralizedGenerated.NGG ~/.julia/packages/GeneralizedGenerated/PV9u7/src/ngg/compact.jl:208
  [6] from_type(#unused#::Type{GeneralizedGenerated.NGG.TypeLevel{Expr, "Buf{23}()"}})
    @ GeneralizedGenerated.NGG ~/.julia/packages/GeneralizedGenerated/PV9u7/src/ngg/typeable.jl:36
  [7] type2model(#unused#::Type{Model{NamedTuple{(:uamp, :vamp, :s1, :s2, :erramp, :u1cp, :v1cp, :u2cp, :v2cp, :u3cp, :v3cp, :errcp), T} where T<:Tuple, GeneralizedGenerated.NGG.TypeLevel{Expr, "Buf{23}()"}, GeneralizedGenerated.NGG.TypeLevel{Module, "Buf{17}()"}}})
    @ Soss ~/.julia/packages/Soss/cix9I/src/core/models/model.jl:29
  [8] macro expansion
    @ ~/.julia/packages/Soss/cix9I/src/primitives/xform.jl:151 [inlined]
  [9] #s65#76
    @ ~/.julia/packages/GeneralizedGenerated/PV9u7/src/closure_conv.jl:132 [inlined]
 [10] var"#s65#76"(Asub::Any, A::Any, B::Any, ::Any, M::Any, _m::Any, _args::Any, _data::Any)
    @ Soss ./none:0
 [11] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any, N} where N)
    @ Core ./boot.jl:571
 [12] xform(m::Soss.ConditionalModel{NamedTuple{(:uamp, :vamp, :s1, :s2, :erramp, :u1cp, :v1cp, :u2cp, :v2cp, :u3cp, :v3cp, :errcp), T} where T<:Tuple, GeneralizedGenerated.NGG.TypeLevel{Expr, "Buf{23}()"}, GeneralizedGenerated.NGG.TypeLevel{Module, "Buf{17}()"}, NamedTuple{(:uamp, :vamp, :s1, :s2, :erramp, :u1cp, :v1cp, :u2cp, :v2cp, :u3cp, :v3cp, :errcp), Tuple{Vector{Float64}, Vector{Float64}, Vector{Symbol}, Vector{Symbol}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Float64}}}, NamedTuple{(:amp, :cphase, :AP, :AZ, :JC, :SM, :AA, :LM, :SP), Tuple{Vector{Float64}, Vector{Float64}, Float64, Float64, Float64, Float64, Float64, Float64, Float64}}})
    @ Soss ~/.julia/packages/Soss/cix9I/src/primitives/xform.jl:22
 [13] top-level scope
    @ REPL[26]:1

The annoying part is that if I define the module in my current session and then just call it everything seems to work fine. The issue only seems to arise when I define the models in a separate package and then bring them into scope with using.

[cscherrer]

Hi @ptiede , thanks for bringing this up! I haven't used Soss in the way you describe, but it's definitely something we want to support.

With some of the metaprogramming tricks we do, we need to be careful things work properly across module and package boundaries. It's hard to say immediately whether this might be a module issue or a precompilation issue, but @rfourquet showed me a nice way of making tests for this case in https://github.com/cscherrer/KeywordCalls.jl/pull/6 . So I'll add something similar to the Soss tests and start digging into it.

@thautwarm any immediate thoughts on this? If not I'll see what I can find and may bug you with GG questions :)

[thautwarm]

I've just found that the new version of GG cannot work with precompilation, so tentatively you cannot precache defining a model.
@ptiede

[thautwarm]

So far you need to disable precompilation.

[ptiede]

@thautwarm thank you! It looks like that fixed everything. Thank you for looking into this so quickly!

[cscherrer]

Thanks @thautwarm ! Just to be sure I understand, does this mean we need to disable precompilation for Soss itself, or for packages that export Soss models?

[thautwarm]

@ptiede You're welcome!

@cscherrer

does this mean we need to disable precompilation for Soss itself, or for packages that export Soss models?

If any Soss model is created at the module top-level, it cannot get correctly precompiled. Besides, making a Soss model at top-level means it's defined statically and not runtime-generated. I'm thinking about how to make things work..

[thautwarm]

Besides, if all Soss models are created at runtime, it's okay to use precompile the downstream module.