Closed williamthome closed 2 months ago
Friendly ping @mworrell @mmzeeman @vkatsuba :slightly_smiling_face:
Just sharing some thoughts here, but I think there are 2 issues to be addressed:
null
and undefined
are conflated.drop_nulls
lacks context, and exhibits feature interaction issues.null
or undefined
or both?We have 2 opposing positions that we have to choose between: is this a library for parsing and building serialised JSON structures or is this a library for mapping between Erlang terms and JSON values?
JSON doesn't really have the notion of undefined
, only null
exists in JSON and as far as I know is typically used to represent the missing value.
Erlang, traditionally, has used undefined
, especially in records, to represent the missing value, but. Elixir on the other hand, standardised on nil
for the missing value.
JavaScript, from which JSON derives, has both undefined
and null
, the first represent the nonexistent value while the later represents the missing value. This is conveniently used by the JSON.stringify
implementation to allow encoding JavaScript objects with both missing and nonexistent values:
> JSON.stringify({a: undefined, b: null, c: 1})
'{"b":null,"c":1}'
As for Arrays, we have this 🤦🏽♂️:
> JSON.stringify([undefined, null, 1])
'[null,null,1]'
The question is: do we want to allow easy construction of JSON objects with nonexistent values?
This would have been a non-question if Erlang had a convenient syntax for building/matching maps with missing values, but it doesn't. So, do we want to use the same pattern as JavaScript and allow a for a special token (undefined
?) that allows us to construct JSON objects with nonexistent values?
drop_nulls
naming, context and feature interactionsThe naming is confusing and is a direct result of (1).
For a start, the feature should probably be scoped to express the fact that this only applies to maps, and not to arrays or plain values, for example: euneus:encode(Term, #{maps => #{skip_values => [undefined]}})
or euneus:encode(Term, #{plugins => [{map, #{skip_values => [undefined]}}]})
(I know this is not how the current API looks like, just sharing some thoughts).
In fact, maybe this shouldn't be a plugin? Maybe plugins should be called codecs instead of plugins? And they should only concern themselves with encoding/decoding values instead of changing the structural encoding logic of lists and maps?
The undefined
is not a official JSON value type - https://datatracker.ietf.org/doc/html/rfc7159#section-1 (C)
JSON can represent four primitive types (strings, numbers, booleans,
and null) and two structured types (objects and arrays).
So, if make sense to focus on point that the only null should be converted as is. At the same time it can be configurable over plugin mechanism which will add more flexible, so, by default type null will converted to atom null, by default undefined fields will be ignore as in JavaScript - and if somebody want to keep undefined fields, well he need to do it by using plugin config stuff.
In https://github.com/zotonic/zotonic, we usually try to avoid null
values from entering the system. So we usually map nulls
from postgres or json to undefined
.
Adding my 2c.
For my use case:
undefined
does not exist in a decode context, while null
does)undefined
does not exist, but null
does.The choice to use null
or undefined
within code is really up to the authors, but in my case, since we're dealing with database queries/results, using null
rather than undefined
keeps things consistent.
From my perspective:
undefined
: this thing does not exist at allnull
: this thing has no valueSo looking at it from an encoder view:
null
should be treated the same as JSON null
undefined
should drop the property from the objectBeing able to drop any properties with a null value is a feature that I would find useful as in my specific case, we do not send properties with null
value over the wire unless there is some weird requirement on the receiver side to receive the property with the null value.
This whole discussion has been around custom plugins to handle the cases, but I believe it's solved more simply as options for encode/decode.
To cover the spectrum:
decode_null_to
= null :: any()decode_drop_values
= [] :: list(any())encode_drop_values
= [undefined] :: list(any())Ending my ramble....
Here's some thoughts...
I agree with @vkatsuba's
by default type null will converted to atom null, by default undefined fields will be ignore as in JavaScript
which is consistent with what I discussed with you over Slack.
This seems to be consistent with "proposal"
euneus:encode(Term, #{maps => #{skip_values => [undefined]}})
if the second argument was actually a default for the function call, and it also seems consistent with
So looking at it from an encoder view: atom
null
should be treated the same as JSONnull
atomundefined
should drop the property from the object
Ok, so I'm planning to release a v2.0
of Euneus in #37 after these comments.
I'm currently busy, but I will explain more as soon as possible.
Nice to hear you all! Thanks for the feedback o/
If you have more suggestions, let me know.
I've been working on this for a while, and last week, I decided to start from scratch and remove all the Euneus code. Now, Euneus is built on top of the new JSON module introduced in OTP 27. There are still things to do, documentation to improve, and the interface is not 100% defined. Feel free to test it out and give any suggestions. I'll close this issue soon when version 2 is released with the changes of the main branch.
I'm closing this issue in favor of the v2.0 release.
Thank you to all of you who contributed here! All of the comment was extremely valuable to this version.
I've announced the version on the Erlang Forums.
Thanks again, and feel free to continue contributing o/
There is a confusing thing about the null literal of JSON in Euneus (and probably in any other Erlang lib that encodes/decodes JSON).
Encode contains the
nulls
option and decode thenull_term
.When encoding,
nulls
is a list of Erlang terms considered to be thenull
literal of the JSON, e.g.:When decoding,
null_term
means what term to be considered as the null literal from JSON in Erlang, e.g.:Both options can be anything that you want.
Why this? Erlang and Elixir do not have a
null
term. What I see as a convention is the use of the atomundefined
for Erlang and:nil
for Elixir to representnull
terms. By default, Euneus considers the Erlang way, so the default for thenulls
option is[undefined]
and fornull_term
isundefined
. So,nulls
anddrop_nulls
plugin can sound strange because them doesn't have/ignore any null term by default.Due to this, I think we cannot use/consider the same Javascript approach, for example:
Using the
drop_nulls
Euneus plugin:@leonardb said:
@paulo-ferraz-oliveira said (translated from Portuguese):
@asabil said:
What I see is: there is no silver bullet.
Maybe some possibilities:
drop_nulls
from the Euneus plugins and let the user define what it considers to be removed from maps and proplists;drop_nulls
(probably not the best name in this case) must contain the terms to be ignored;Please help me by answering my question below: What do you see as the best approach to consider null, undefined, nil, etc from Erlang to JSON and null from JSON to Erlang?