Quick and dirty support for array-to-field binding

[omartijn]

I have a weird serialization format I need to support, where I need to (de)serialize a JSON array with different types to a struct. It's something like this:

{"x": [1, "abc"]}

Now, I want to deserialize the array to a struct like this:

struct from_array {
    std::uint64_t number;
    std::string string;

    template <io_type>
    void json_io(io_type& io) {
        io& json_dto::optional_array(number, string);
    }
};

That works with the code I've added to this PR. I know, the format is wonky, why it is stored inside an array and not an object I cannot say, but I need to be able to decode this. Maybe it's useful to bake into json_dto?

[eao197]

@omartijn , thanks for PR.

Unfortunately, I have no time to handle your PR this week, so I hope to return to it some time later. It is a open question for me is it worth to have such functionality in the library and I want to discuss it with colleguas.

I'm sorry for that delay.

[omartijn]

@eao197 No rush there. I'm using it myself right now, with a simple helper binding outside the json_dto namespace, so I can just use it like helpers::optional_array(...) which works well enough. If you see no value in adding it json_dto itself that's totally understandable - like I said: The JSON structure used is wonky.

That's also why I have not properly made it to use the different policies to for readers/writers, nulls and optionals and all. It works for me, and it would be a waste of time to implement everything "properly" if it's not going to be merged upstream.

[eao197]

Hi!

I think that such a feature could be a good addition to the library. But, I'm afraid, not in the proposed form.

Now I'm thinking about something like this:

struct nested {
  int a;
  std::string b;
};

struct outer {
  nested x;

  template<typename Json_Io>
  void json_io(Json_Io & io) {
    io & json_dto::mandatory("x",
      json_dto::tuple_inside_array(x).with(&nested::a).with(&nested::b));
  }
};

It may allow to have a "normal" json_io for nested struct:

struct nested {
  int a;
  std::string b;

  template<typename Json_Io>
  void json_io(Json_Io & io) {
    io & json_dto::mandatory("a", a)
      & json_dto::mandatory("b", b);
  }
};
...
struct another_outer {
  nested x;

  template<typename Json_Io>
  void json_io(Json_Io & io) {
    io & json_dto::mandatory("x", x);
  }
};

It could also support custom Reader_Writer for tuple's items:

struct custom_floating_point_reader_writer {... /* See README.md for an implementation example */ };

struct nested {
  int a;
  std::string b;
  float c;
};

struct outer {
  nested x;

  template<typename Json_Io>
  void json_io(Json_Io & io) {
    io & json_dto::mandatory("x",
      json_dto::tuple_inside_array(x)
        .with(&nested::a)
        .with(&nested::b)
        .with(custom_floating_point_reader_writer, &nested::c));
  }
};

But it's just a sketch. I don't know yet can it be implemented at all (or will it be a good solution).

[omartijn]

I think tuple_inside_array is an OK name for this feature. I'm not quite sure how the "normal" json_io would work. Are the field names synthesized by json_dto? Since it's a plain array, there are no field names of course!

The way you showed it now, with a "nested" struct is not the way I imagine using it, but I guess if you made it like this you could also pass *this and just use regular fields as well, so that shouldn't be an issue I think.

If we are thinking about maximum flexibility, I can imagine scenarios where you'd want some of the array elements to be mandatory, while some are optional, and of course you could argue that it's OK for the whole array not to be there, but if it's there, it needs to contain a certain number of elements.

[eao197]

Hi!

I've made a first attempt to do something in this direction: https://github.com/Stiffstream/json_dto/blob/issue-19-v03/dev/test/inside_array/main.cpp But I don't like the current result. Maybe the right way will be to go in a different direction, something like this:

struct inner {
  int x;
  std::string y;

  template<typename Io>
  void json_io(Io & io) {
    json_dto::inside_array(io)
      .with(simple_int_reader_writte{}, x)
      .with(y);
  }
};

struct outer {
  inner a;

  template<typename Io>
  void json_io(Io & io) {
    io & json_dto::mandatory("a", a);
  }
};

This approach could allow to specify validators for a in outer::json_io. It should also allow to use json_dto::optional, json_dto::mandatory_with_null_as_default and so on.

Another variant is to write something like that:

struct inner {
  int x;
  std::string y;

  template<typename Io>
  void json_io(Io & io) {
    io[
      json_dto::array_item(simple_int_reader_writte{}, x),
      json_dto::array_item(y)
    ];
  }
};

I have no time to find a good solution before the New Year Eve and now I'm on a small vacation and I'm panning to return to this task on the next week.

[omartijn]

I do hope you aren't meaning the coming New Years' Eve - which is almost a full year off! :smiley: Please enjoy your holiday, though.

I'm thinking of a generic approach where we can use all the customization points (e.g. mandatory, on null, etc) that we have also for arrays. I guess the key to that is variants of binder_read_from_implementation_t and binder_write_to_implementation_t. We could then have the overloaded operator& check the policy to see if the field is an array type and then keep an index (so we know the index within the array).

[eao197]

Hi! Here is another attempt.

A very simple example looks like this: https://github.com/Stiffstream/json_dto/blob/e62c1540625bdffd7ea77d5d24107d350cfebdf4/dev/test/inside_array/main.cpp#L14-L72

One of drawbacks is the repetitions of simple_test_t:

json_dto::inside_array<simple_nested_t>(
        json_dto::array_member( &simple_nested_t::m_a ),
        json_dto::array_member( &simple_nested_t::m_b ),
        json_dto::array_member( &simple_nested_t::m_c ) ),

but I've kept this version because it has type safety. For example, you can write something like that:

struct first_inner {
  int m_a;
  int m_b;
};
struct second_inner {
  int m_a;
  int m_b;
  int m_c;
};

struct outer {
  first_inner m_first;
  second_inner m_second;

  template<typename Json_Io>
  void json_io(Json_Io & io) {
    io & json_dto::mandatory(
        json_dto::inside_array(
          json_dto::array_member(m_first.m_a),
          json_dto::array_member(m_first.m_b),
          json_dto::array_member(m_second.m_c)),
      "x", m_second);
  }
};

One important thing is not implemented in the current draft: validators. It's impossible to specify a validator in json_dto::array_member for now. But I hope it can be added easily.

[omartijn]

That looks promising, though IIUC it does not allow to have, say, the first two elements in the array mandatory, and the third one and after optional.

Not sure what the interface for that would be. You wouldn't want to have something like mandatory and optional within the inside_array, as that leads to nonsense - that way you could write a mandatory after an optional which is clearly ridiculous. I think we should be able to signal to the inside_array how many elements we require at a minimum.

[eao197]

That looks promising, though IIUC it does not allow to have, say, the first two elements in the array mandatory, and the third one and after optional.

From my point of view, having "optional" in inside_array is possible only if first N are mandatory (and only mandatory) and remaining are optional.

I will think about something like that:

json_dto::inside_array<simple_nested_t, json_dto::mandatory_items<2>>(
        json_dto::array_member( &simple_nested_t::m_a ),
        json_dto::array_member( &simple_nested_t::m_b ),
        json_dto::array_member( &simple_nested_t::m_c ) ),

where json_dto::mandatory_items<2> tells that the first two are mandatory, but all other are optional.

[eao197]

One of drawbacks is the repetitions of simple_test_t: but I've kept this version because it has type safety.

Another issue with this decision: it's impossible to use inside_array for (de)serializing tuples.

Maybe we should have something like that:

struct demo {
  std::tuple<int, std::string, float> m_items;
  ...
  template<typename Json_Io>
  void json_io(Json_Io & io) {
    io & json_dto::mandatory(
        json_dto::inside_array(
          json_dto::array_member(std::get<0>(m_items)),
          json_dto::array_member(std::get<1>(m_items)),
          json_dto::array_member(std::get<2>(m_items))),
       "items", m_items)
     ...
  }
};

[eao197]

Another issue with this decision: it's impossible to use inside_array for (de)serializing tuples.

I think it's possible to do something like that:

struct demo {
  std::tuple<int, std::string, float> m_items;
  ...
  template<typename Json_Io>
  void json_io(Json_Io & io) {
    io & json_dto::mandatory(
        json_dto::inside_array<decltype(demo::m_items)>( // json_dto can detect that it's a tuple.
          json_dto::array_member(json_dto::tuple_item<0>{}), // json_dto can check the index of tuple_item.
          json_dto::array_member(json_dto::tuple_item<1>{}),
          json_dto::array_member(json_dto::tuple_item<2>{}),
       "items", m_items)
     ...
  }
};

And it could provide saficient type safety.

But I doubt that it's worth the complexity of implementation.

[eao197]

Another intermediate step :)

A bit different naming. No pointers to members. But std::tuple can be supported (manually):

struct tuple_holder_t
{
    std::tuple<int, int, std::string, int> m_x;

    tuple_holder_t()
        : m_x{ 0, 1, "zero", 2 }
    {}

    template< typename Json_Io >
    void
    json_io( Json_Io & io )
    {
        io
            & json_dto::mandatory(
                    json_dto::inside_array::reader_writer(
                        json_dto::inside_array::member( std::get<0>(m_x) ),
                        json_dto::inside_array::member(
                            simple_int_reader_writter_t{}, std::get<1>(m_x) ),
                        json_dto::inside_array::member( std::get<2>(m_x) ),
                        json_dto::inside_array::member( std::get<3>(m_x) ) ),
                    "x", m_x );
    }
};

There is also at_least limiter for numbers of members of an array:

struct at_least_checker_one_t
{
    int m_x1{};
    int m_x2{};
    int m_x3{};
    int m_x4{};

    template< typename Json_Io >
    void
    json_io( Json_Io & io )
    {
        io
            & json_dto::mandatory(
                    json_dto::inside_array::reader_writer<
                            json_dto::inside_array::at_least<2> >(
                        json_dto::inside_array::member( m_x1 ),
                        json_dto::inside_array::member( m_x2 ),
                        json_dto::inside_array::member( m_x3 ),
                        json_dto::inside_array::member( m_x4 ) ),
                    "x", *this );
    }
};

By default a T{} is used for initialization of missed member. For example:

const char * json_str =
    R"({
        "x":[ 1, 2 ]
    })";

at_least_checker_one_t r;
r.m_x1 = 33;
r.m_x2 = 34;
r.m_x3 = 35;
r.m_x4 = 36;

json_dto::from_json( json_str, r );

REQUIRE( 1 == r.m_x1 );
REQUIRE( 2 == r.m_x2 );
REQUIRE( 0 == r.m_x3 ); // It's because of `r.m_x3 = int{}` during deserialization.
REQUIRE( 0 == r.m_x4 ); // It's because of `r.m_x4 = int{}` during deserialization.

But it seems that we have to have something like that -- json_dto::inside_array::member_with_default(field, default_value) -- where default_value will be used on deserialization if the member is not present in the array (only on deserialization, on serialization the field will be stored anyway even if it has default_value).

[eao197]

Hi!

I hope that the branch issue-19-v05 now contains completed version of support for such functionality. Including several new examples and a brief description in the README: https://github.com/Stiffstream/json_dto/blob/ee862ee579f3523c20d16983939bf287fd532748/README.md#representing-several-fields-inside-an-array

If there won't be any significant flaws found I'll release it the next week (including fixes for #23).

[eao197]

Hi! I'll close this PR because version 0.3.3 contains a solution for this problem.