Optionally Write Keys at Runtime (while using Partials)

[pawcam]

I've been using partial writes successfully for some time now. What I'd like to do is conditionally hide certain keys when writing. There doesn't seem to be a way to combine [glz](glz::write_json<partial>) with glz::opts = {.skip_null_members=true}. In addition to this, I've also had issues compiling due to type conflicts with returning nullptr conditionally in combination with other types (const char* and std::sting_view). For example my glaze::meta<> specialization for writing a certain field just returns an empty string but still serializes the key in the json (which makes sense since it's nullptr, technically undefined behavior)

static constexpr auto const write_head = [](const auto& s)
  {
    if (!StringUtils::isEmpty(s.m_pSerializeData->szSymbol))
    {
      return s.m_pSerializeData->szSymbol;
    }
    else if (!s.bHeadSet)
    {
      return nullptr;
    }
    else
    {
      return FieldHelper::symbolFromInstrument(); // returns some string
    }
  };

When I try to add more partials my understanding is that there is some internal hashing limited by the number of constexpr the compiler supports (which I've hit due to the large set of keys I need to support). This makes sense considering all the keys need to be hashed at compile time.

partial_for_message_A = json_ptrs("/account", "size", /*....list of 20 keys*/)
partial_for_message_B = json_ptrs("/account", "size", /*....list of 20 keys*/)

// some modified array based on A
partial_for_modified_message_A = json_ptrs("/account", "size", /*....list of 20 keys*/)

Im still working through the best way to do this, but is there an efficient way to hide/show keys based on runtime data within the glaze library? I know this technically goes against the spirit of what the library is for (compile time key mapping), but if there were a mechanism to hide already(compile-time) known keys at runtime, I think it would make things a lot more flexible and I wouldn't need to try as many workarounds. We could implement some sort of tag dispatching to inform the call sit of glz::write_json to use a different templated version of our struct, but I haven't fully though this through.

This seems to be the trickiest part of our use case, so if there's an easy way to do it using your library it would really tie everything together. We have a lot of conditional fields based on runtime data that go into consideration under larger buckets. The large buckets (Message A, B, and C) are all generally the same set of fields, so partials work great for them. It's the more granular keys (if Message A is for this type of event, then don't/do send this key, etc.) that become challenging.

Thanks again for all the help.

[stephenberry]

Thanks for explaining your problem in detail. We can still have compile time hash tables with runtime inclusion/exclusion, since we know our potential keys at compile time.

The problem with runtime JSON pointers to keys is that this induces runtime sorting and grouping, which is a significant cost. But, I'm sure you're looking for flexibility and not peak performance.

Better performance can be achieved if we don't use JSON pointers and only allow top level keys.

Do you find yourself using nested partial writing? Or, would you be satisfied with only top level keys in an object being specified?

Potential API:

std::vector<std::string> keys = { "key_one", "key_two" };
glz::write_partial_json(keys, object, buffer);

[pawcam]

We're only writing top level keys for the foreseeable future, so this would be a nice enhancement.

That api is pretty much what I want to do. I'd love to keep the compile time key performance, but It would be nice to set flags in my struct that indicate which known compile time keys that should be ignored during serialization (you have a flags() method available in glz::meta but Im not sure it solves my use case):

static constexpr auto value = flags("my-conditional-key", &T::bSerializeConditionalKey); // this seems to only work for the field bSerializeConditionalKey itself, not another field with a real value.

Something like:

struct myData
{
  bool bField1Flag{false};
  bool bField2Flag{false};

 char someField1[]
 char someField2[]
}

template <>
struct meta<myData>
{
using T = myData;

  static constexpr auto value = object("field-1", , glz::custom<read_field_1  , write_field1
, true                     // always read
, &T::bFieldFlag1 // runtime conditionally write
>.
}