Deserialize optional fields (Union types)

ianfiske commented 4 months ago

I am trying to deserialize user config data that has fields with multiple possible option types, as implemented with a type-union. This seems to not be supported. Are there plans to support this? Or any alternatives you might suggest?

Here's an example:

using Serde

struct B1
    y::Float64
end

struct B2
    x::Float64
    y::Float64
end

struct A
    x::Union{B1, B2}
end

json = """
{
    "x": {
        "x": 1.0,
        "y": 2.0
    }
}
"""

deser_json(A, json)

gives the error

julia> deser_json(A, json)
ERROR: WrongType: for 'A' value 'Dict{String, Any}("x" => 1.0, "y" => 2.0)' has wrong type 'x::Dict{String, Any}', must be 'x::Union{B1, B2}'
Stacktrace:
 [1] eldeser(structtype::Type, elmtype::Type, key::Symbol, val::Dict{String, Any})
   @ Serde ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:510
 [2] deser(::Serde.CustomType, ::Type{A}, data::Dict{String, Any})
   @ Serde ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:529
 [3] deser
   @ ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:164 [inlined]
 [4] to_deser(::Type{A}, x::Dict{String, Any})
   @ Serde ~/.julia/packages/Serde/ShhQA/src/De/De.jl:87
 [5] deser_json(::Type{A}, x::String; kw::@Kwargs{})
   @ Serde.DeJson ~/.julia/packages/Serde/ShhQA/src/De/DeJson.jl:34
 [6] deser_json(::Type{A}, x::String)
   @ Serde.DeJson ~/.julia/packages/Serde/ShhQA/src/De/DeJson.jl:33
 [7] top-level scope
   @ ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:573

caused by: MethodError: no method matching fieldnames(::Type{Union{B1, B2}})

Closest candidates are:
  fieldnames(::Core.TypeofBottom)
   @ Base reflection.jl:170
  fieldnames(::Type{<:Tuple})
   @ Base reflection.jl:172
  fieldnames(::UnionAll)
   @ Base reflection.jl:169
  ...

Stacktrace:
  [1] _field_types(::Type{Union{B1, B2}})
    @ Serde ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:479
  [2] deser(::Serde.CustomType, ::Type{Union{B1, B2}}, data::Dict{String, Any})
    @ Serde ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:524
  [3] deser(::Type{Union{B1, B2}}, data::Dict{String, Any})
    @ Serde ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:164
  [4] deser(::Type{A}, ::Type{Union{B1, B2}}, data::Dict{String, Any})
    @ Serde ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:156
  [5] eldeser(structtype::Type, elmtype::Type, key::Symbol, val::Dict{String, Any})
    @ Serde ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:504
  [6] deser(::Serde.CustomType, ::Type{A}, data::Dict{String, Any})
    @ Serde ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:529
  [7] deser
    @ ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:164 [inlined]
  [8] to_deser(::Type{A}, x::Dict{String, Any})
    @ Serde ~/.julia/packages/Serde/ShhQA/src/De/De.jl:87
  [9] deser_json(::Type{A}, x::String; kw::@Kwargs{})
    @ Serde.DeJson ~/.julia/packages/Serde/ShhQA/src/De/DeJson.jl:34
 [10] deser_json(::Type{A}, x::String)
    @ Serde.DeJson ~/.julia/packages/Serde/ShhQA/src/De/DeJson.jl:33
 [11] top-level scope
    @ ~/.julia/packages/Serde/ShhQA/src/De/Deser.jl:573

gryumov commented 4 months ago

@ianfiske Hi! Here's a quick solution I can suggest, but just remember to prioritize the types correctly.

using Serde

struct B1 
    y::Float64
end

struct B2
    x::Float64
    y::Float64
end

struct A
    x::Union{B1, B2}
end

function Serde.deser(::Type{<:A}, ::Type{<:Union{B1,B2}}, v)
    try
        Serde.deser(B2, v)
    catch
        Serde.deser(B1, v)
    end
end

json1 = """
{
    "x": {
        "x": 1.0,
        "y": 2.0
    }
}
"""

julia> deser_json(A, json1)
A(B2(1.0, 2.0))

json2 = """
{
    "x": {
        "y": 2.0
    }
}
"""

julia> deser_json(A, json2)
A(B1(2.0))

gryumov commented 4 months ago

If you provide me with more real-world cases, or suggest the expected interface, we can consider how to support this within the library.

gryumov commented 4 months ago

Take a look at this https://github.com/bhftbootcamp/Serde.jl/issues/37#issuecomment-2091038501, it might be relevant to you.

ianfiske commented 4 months ago

Thank you for the suggestion @gryumov ! That seems like it should work. Here's an example that more closely resembles my use-case. There are a number of data sources. Each data source (timeseries) can either be an Int (correspondong to some id in a db) or a structure that describes how to locate the file locally.

struct LocalFileData
    file_path::String
    column_name::String
end

struct Data
    timeseries1::Union{LocalFileData,Int}
    timeseries2::Union{LocalFileData,Int}
end

json = """
{
    "timeseries1": 13235,
    "timeseries2": {
        "file_path": "/path/to/csv"
        "column_name": "A"
    }
}
"""

ianfiske commented 4 months ago

Accidentally "closed" there... thought it was a drop-down option along with the main button.

NeroBlackstone commented 4 months ago

I think this feature is easy to implement, just select the type that best matches the field when deserializing

kapple19 commented 3 weeks ago

Commenting to also note my interest.

In addition to the above use cases, I also have some fields as Union{Int, Nothing} as an example. Works really well in JSON3.jl. But would be nice to have a multiformat de/serialiser like Serde.jl

bhftbootcamp / Serde.jl

Deserialize optional fields (Union types) #51