Dealing with optional fields in maps

Problem

Occasionally, you'll want to decode something into a map() where certain fields are optional. Perhaps your decoder doesn't have enough information to provide sane defaults, or you wish to mirror the things your API accepts fairly closely and deal with defaulting in another layer using maps:get/3.

Current workarounds

This can currently be worked around in a few ways.

Let's imagine we have a person/0 type like so:

-type person() :: #{ name   := binary()
                   , weight => pos_integer()
                   }.

And our JSON is allowed to leave out the weight field entirely.

One option is providing a default:

-spec person_decoder() -> dj:decoder(person()).
person_decoder() ->
  dj:to_map(#{ name => dj:field(name, dj:binary())
             , weight => dj:one_of(dj:field(name, dj:binary()), 1)
             }).

This isn't great - both the JSON and our person/0 type allow leaving out the weight field, yet we enforce it to be available.

Another option is using a "magical" constant and filtering:

-spec person_decoder() -> dj:decoder(person()).
person_decoder() ->
  Decoder = dj:to_map(#{ name => dj:field(name, dj:binary())
                       , weight => dj:one_of(dj:field(weight, dj:pos_integer()), 1)
                       }),
  Filter = fun (_, not_provided) -> false;
               (_, Val)          -> true
           end,
  dj:map(fun (M) -> maps:filter(Filter, M) end, Decoder).

Other variations involving filtering are possible. These workarounds are definitely bulky and all but elegant.

Proposal

Some way(s) to allow specifying optional properties would be nice. I see three contenders.

Option 1: Add an `optional/1` helper

-spec person_decoder() -> dj:decoder(person()).
person_decoder() ->
  dj:to_map(#{ name => dj:field(name, dj:binary())
             , weight => dj:optional(dj:field(weight, dj:pos_integer()))
             }).

Essentially it would ignore missing_field errors, instead removing the "offending" key from the resulting map.

👍 Fairly easy API
👎 Additional nesting add additional "depth" to decoders, which comes at a price
👎 Not clear how this should behave outside to_map

Option 2: Allow specifying optional fields

-spec person_decoder() -> dj:decoder(person()).
person_decoder() ->
  dj:to_map(#{ name => dj:field(name, dj:binary())
             , weight => dj:field(weight, dj:pos_integer())
             }
            , [weight]
            ).

👍 Similarly easy API
👎 Requires separating the map spec from the optional fields -> easy to make typos

Option 3: Remove the ability to take a `map()` in favour of a tuple-list

-spec person_decoder() -> dj:decoder(person()).
person_decoder() ->
  dj:to_map([ {name, dj:field(name, dj:binary())}
            , {weight, optional, dj:field(weight, dj:pos_integer())}
            ]).

👍 API already exists, but is "extended"
👎 I quite like the ability to pass a map(), but don't want to create a feature-imparity between the different ways to call to_map()

For people who like visually laying things out, we could instead expand the spec to be something like this:

-spec to_map(MapSpec) -> decoder(map()) when
    MapSpec   :: [FieldSpec],
    FieldSpec :: {Field, Decoder} | {Field, Req, Decoder},
    Field     :: term(),
    Decoder   :: decoder(term()),
    Req       :: required | optional.

So the following decoder would be equivalent:

-spec person_decoder() -> dj:decoder(person()).
person_decoder() ->
  dj:to_map([ {name, required, dj:field(name, dj:binary())}
            , {weight, optional, dj:field(weight, dj:pos_integer())}
            ]).

CC @kwrooijen @sgillis

Thinking about this some more, Option 3 can be combined with either Option 1 or Option 2, too.

Option 1+3

-spec optional(Dec) -> {optional, Dec} when Dec :: decoder(T).

-spec to_map(MapSpec) -> decoder(MapResult) when
    MapSpec :: #{Key := Dec} | [PropListEntry],
    PropListEntry :: {Key, Dec} | {Key, optional, decoder(T)},
    Dec :: decoder(T) | {optional, decoder(T)},
    MapResult :: #{Key => T}.

This does lead to quite a few equivalent ways of specifying a decoder with an optional field, and I'm not sure if that's a great idea:

dj:to_map(#{foo => dj:optional(dj:field(foo, dj:binary()))}),
dj:to_map(#{foo => {optional, dj:field(foo, dj:binary())}}),
dj:to_map([{foo, dj:optional(dj:field(foo, dj:binary()))}]),
dj:to_map([{foo, optional, dj:field(foo, dj:binary())}]).

Option 2+3

-spec to_map(MapSpec) -> decoder(MapResult) when
    MapSpec :: #{Key := Dec} | [PropListEntry],
    PropListEntry :: {Key, Dec} | {Key, Optionality, Dec},
    Optionality :: optional | required,
    Dec :: decoder(T),
    MapResult :: #{Key => T}.

-spec to_map(MapSpec, OptionalFields) -> decoder(MapResult) when
    MapSpec :: #{Key := Dec} | [{Key, Dec}],
    Dec :: decoder(T),
    OptionalFields :: [Key],
    MapResult :: #{Key => T}.

Then again, this still allows expressing the same decoder in three equivalent ways:

dj:to_map(#{foo => dj:field(bar, dj:binary())}, [foo]),
dj:to_map([{foo, dj:field(bar, dj:binary())}], [foo]),
dj:to_map([{foo, optional, dj:field(bar, dj:binary())}]).

Considering all of the above, I'm starting to lean toward only doing Option 2:

it leads to shorter decoders
it appears fairly clear
it expresses that the field in the resulting map is optional, rather than whatever field is being decoded, which doesn't strictly have to be a field!

truqu / dj