[INFO] Pointers towards deriving/generating julia data types in rust to call julia functions

[nabeel99]

Hey Op, Thank you for working hard on this brilliant crate, i recently had a usecase where i wanted to use a certain julia library in rust : The data types of the library in question : https://github.com/bcc-research/CFMMRouter.jl/blob/1cfd60f70584dfd979351483d30f1bf20bbb41c7/src/router.jl

If you could tell the general way to derive julia types into rust data types, that would be helpful since i cdnt find any example nor somewhere in the docs on how the jlrs derive macro is used to convert a julia d.t to a rust one. Also i think i read that whatever way used to automatically convert julia d.t to rust has the limitation that it dsnt work with generic julia d.t if so ,is there a way around it by manually impl the types if so what is the intuition like ?

[Taaitaaiger]

You can add the JlrsCore package, the Reflect module has a reflect function that takes a vector of type objects and generates Rust implementations that derive all applicable traits.

https://github.com/Taaitaaiger/JlrsCore.jl

[nabeel99]

hey thanks for the quick reply i followed your advice and managed to generate rust equivalent types for all types except one, Approach : go to the package and copy all structs into one file and use the reflect package to get its rust equivalent types if there is a better more standard approach than doing this do let me know since this is my first time using julia i dont know if one can just provide a julia package and have all its type automatically derived or smth. i went with the naive painstaking approach of copying all structs from all the *.jl files into one

Problem

struct GeometricMeanTwoCoin{T} <: CFMM{T}
     R::Vector{T}
    γ::T
    Ai::Vector{Int}
    w::SVector{2,T}
    function GeometricMeanTwoCoin(R, w, γ, idx)
        γ_T, idx_uint, T = two_coin_check_cast(R, γ, idx)

        return new{T}(
            MVector{2,T}(R),
            γ_T,
            MVector{2,UInt}(idx_uint),
            SVector{2,T}(w),
        )
    end
end

it fails with the error : ERROR: LoadError: UndefVarError: SVector not defined

[nabeel99]

fixed the above issue by importing StaticArrays but structs with Svector now give out the following error :

ERROR: LoadError: MethodError: no method matching extractdeps!(::Dict{DataType, Set{DataType}}, ::Core.TypeofVararg, ::Dict{Type, JlrsCore.Reflect.Layout})

Closest candidates are:
  extractdeps!(::Dict{DataType, Set{DataType}}, ::Type, ::Dict{Type, JlrsCore.Reflect.Layout})
   @ JlrsCore ~/.julia/packages/JlrsCore/wtMmW/src/Reflect.jl:647

Stacktrace:
 [1] extracttupledeps_notconcrete!(acc::Dict{DataType, Set{DataType}}, type::DataType, layouts::Dict{Type, JlrsCore.Reflect.Layout})
   @ JlrsCore.Reflect ~/.julia/packages/JlrsCore/wtMmW/src/Reflect.jl:610
 [2] extractdeps!(acc::Dict{DataType, Set{DataType}}, type::Type, layouts::Dict{Type, JlrsCore.Reflect.Layout})
   @ JlrsCore.Reflect ~/.julia/packages/JlrsCore/wtMmW/src/Reflect.jl:671
 [3] extractdeps!(acc::Dict{DataType, Set{DataType}}, type::Type, layouts::Dict{Type, JlrsCore.Reflect.Layout})
   @ JlrsCore.Reflect ~/.julia/packages/JlrsCore/wtMmW/src/Reflect.jl:678
 [4] extractdeps!
   @ ~/.julia/packages/JlrsCore/wtMmW/src/Reflect.jl:702 [inlined]
 [5] reflect(types::Vector{UnionAll}; f16::Bool, internaltypes::Bool)
   @ JlrsCore.Reflect ~/.julia/packages/JlrsCore/wtMmW/src/Reflect.jl:191
 [6] reflect(types::Vector{UnionAll})
   @ JlrsCore.Reflect ~/.julia/packages/JlrsCore/wtMmW/src/Reflect.jl:185
 [7] top-level scope
   @ ~/Desktop/CFMMRouter.jl/examples/generateTypes.jl:130

[Taaitaaiger]

Ah, right, varargs are not supported yet. I'll need to check what can be done.

Also, you should use the package for which you want to generate bindings to ensure the whole path is known, not copy-and-paste implementations. The reason is that when jlrs needs to access the type object at runtime, the first time it is looked up based on its fully qualified name.

[nabeel99]

I did import the package but forgive my lack of understanding of julia, how would i make my julia script use the data type from the package directly ? A small example or pseudo code would help understand. To be clear i dont understand your point of "use the package to derive the types for" what does using the package w.r.t deriving mean here ? Go to the main module of the julia package in question and write the script there ? Or importing it as a standalone qorks too ?

[Taaitaaiger]

i went with the naive painstaking approach of copying all structs from all the *.jl files into one

Basically this creates duplicates of these types in the Main module. Instead, you want to want to do something like this:

using Foo
using JlrsCore.Reflect

reflect([TypeInFoo, ...])

[nabeel99]

@Taaitaaiger On the issue of variadic argd, correct me if am wrong, if one is able to convert julia code into compatible abi could one not circumvent the variadic problem as bindgen will prolly be able to create a interfaces which is callable in rust, there is a high chance am completely wrong with my understanding of ffi though so feel free to correct me.

[Taaitaaiger]

The variadic problem here is that some of the type parameters are variadic, not that a function is variadic. It originates here: https://github.com/JuliaArrays/StaticArraysCore.jl/blob/main/src/StaticArraysCore.jl

[nabeel99]

i went with the naive painstaking approach of copying all structs from all the *.jl files into one

Basically this creates duplicates of these types in the Main module. Instead, you want to want to do something like this:

using Foo
using JlrsCore.Reflect

reflect([TypeInFoo, ...])

aha so if i import a certain julia package,the types are exposed and i can just pass there names to the reflect array ? instead of recopying them to the script , hope i got this correct.

[nabeel99]

@Taaitaaiger where would one create the base types that exist in julia for example "zeros(1), zeros(2)" etc say i want to call a function that expects either zeros(1) or zeros(2) or for example something as simple as a array of some type T is it possible to create that with jlrs and pass it into the function ?

[nabeel99]

closing this and opening other issue since the question initially asked was answered, its better to create another issue so others can find answers more easily if they face the same problem.

[Taaitaaiger]

In response to that last question: zeros can take a DataType as its first argument and a size as its second. For example, calling zeros(Float32, 3):

let ty = DataType::float32_type(&frame).as_value();
let n = Value::new(&mut frame, 5);
let func = Module::base().global(&mut frame, "zeros").unwrap()
func.call2(&mut frame, ty, n)

[Taaitaaiger]

I've also looked at the implementation of SVector and I'm afraid I don't think I can support generating bindings for them. You can do so manually, but that isn't a beginner-friendly task IMO.

Value does provide methods like Value::get_field and Value::get_nth_field that let you access its fields without having to know the layout.