Yaml configuration with map of maps

[MeanSquaredError]

It seems that figcone has trouble handling a map of maps, e.g the following config structure

struct config_struct
{
    using auth_map = std::unordered_map<std::string, std::string>;
    using servers_map = std::unordered_map<std::string, auth_map>;
    servers_map servers;
};

Generates the following error:

In file included from /usr/local/projects/private/myproj/source/config/config_reader.cpp:5:
/usr/local/include/figcone/configreader.h: In instantiation of ‘void figcone::ConfigReader::loadField(TCfg&, TField&, std::string_view) [with TCfg = config_struct; TField = std::unordered_map<std::__cxx11::basic_string<char>, std::unordered_map<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> > >; std::string_view = std::basic_string_view<char>]’:
/usr/local/include/figcone/configreader.h:394:19:   required from ‘void figcone::ConfigReader::loadStructure(TCfg&, std::index_sequence<Ints ...>) [with TCfg = config_struct; long unsigned int ...indices = {0, 1, 2, 3}; std::index_sequence<Ints ...> = std::integer_sequence<long unsigned int, 0, 1, 2, 3>]’
/usr/local/include/figcone/configreader.h:406:22:   required from ‘void figcone::ConfigReader::loadStructure(TCfg&) [with TCfg = config_struct]’
/usr/local/include/figcone/configreader.h:422:26:   required from ‘TCfg figcone::ConfigReader::readConfig(const figcone::TreeNode&) [with TCfg = config_struct]’
/usr/local/include/figcone/configreader.h:311:53:   required from ‘std::conditional_t<(rootType == figcone::RootType::SingleNode), TCfg, std::vector<TCfg> > figcone::ConfigReader::read(std::istream&, figcone::IParser&) [with TCfg = config_struct; figcone::RootType rootType = figcone::RootType::SingleNode; std::conditional_t<(rootType == figcone::RootType::SingleNode), TCfg, std::vector<TCfg> > = config_struct; std::istream = std::basic_istream<char>]’
/usr/local/include/figcone/configreader.h:90:36:   required from ‘std::conditional_t<(rootType == figcone::RootType::SingleNode), TCfg, std::vector<TCfg> > figcone::ConfigReader::readFile(const std::filesystem::__cxx11::path&, figcone::IParser&) [with TCfg = config_struct; figcone::RootType rootType = figcone::RootType::SingleNode; std::conditional_t<(rootType == figcone::RootType::SingleNode), TCfg, std::vector<TCfg> > = config_struct]’
/usr/local/include/figcone/configreader.h:134:40:   required from ‘std::conditional_t<(rootType == figcone::RootType::SingleNode), TCfg, std::vector<TCfg> > figcone::ConfigReader::readYamlFile(const std::filesystem::__cxx11::path&) [with TCfg = config_struct; figcone::RootType rootType = figcone::RootType::SingleNode; std::conditional_t<(rootType == figcone::RootType::SingleNode), TCfg, std::vector<TCfg> > = config_struct]’
/usr/local/projects/private/myproj/source/config/config_reader.cpp:10:53:   required from here
/usr/local/include/figcone/configreader.h:357:100: error: static assertion failed: Dict value type must be readable from stringtream or registered with StringConverter
  357 |                     detail::canBeReadAsParam<typename sfun::remove_optional_t<TField>::mapped_type>(),
      |                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~
/usr/local/include/figcone/configreader.h:357:100: note: ‘figcone::detail::canBeReadAsParam<std::unordered_map<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> > >()’ evaluates to false

[MeanSquaredError]

Trying to parse YAML to a map with string keys and structure values fails too.

struct config_struct
{
    struct mystruct
    {
        bool myval;
    };
    std::unordered_map<std::string, mystruct> myval2;
};

The above fails again with

/usr/local/include/figcone/configreader.h:357:100: error: static assertion failed: Dict value type must be readable from stringtream or registered with StringConverter
  357 |                     detail::canBeReadAsParam<typename sfun::remove_optional_t<TField>::mapped_type>(),
      |                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~
/usr/local/include/figcone/configreader.h:357:100: note: ‘figcone::detail::canBeReadAsParam<config_struct::mystruct>()’ evaluates to false

[kamchatka-volcano]

Hi, thanks. The second case seems reasonable, I'm not sure how I missed it. To think of it, it makes more sense to support it than the currently supported map of arbitrary parameters (FIGCONE_DICT). It seems that it's not impossible to implement and it should work pretty much the same as FIGCONE_NODELIST. It's a pretty large feature though, and I'm currently not working much on side projects, so I'm not sure when I'll find time to implement it.

The first case is different, and I doubt that I will support mapping of config structure to nested containers. You can register the value type of a map as a user-defined type with StringConverter<std::unordered_map<std::string, std::string>> and implement its parsing by yourself, but I doubt that it's useful)

[MeanSquaredError]

currently supported map of arbitrary parameters (FIGCONE_DICT)

Do you mean that currently FIGCONE_DICT supports maps where the value is a structure? Or does it require the value to always be a primitive, like std::string, int, bool, etc.?

Regarding the first case (nested containers), I have a YAML configuration which contains servers, index by server_id (string) and the config entry for each server contains a list of authentication parameters, where each server can have a different list of connection parameters.

A sample config would be:

{
    "server_1": {
        "server_1_param_1": "value",
        "server_1_param_2": "value",
        "server_1_param_3": "value",
    },
    "server_2": {
        "server_2_param_1": "value",
        "server_2_param_2": "value",
    },
    ...
}

So using two nested maps seems like a natural way to handle this kind of configuration. The outer map has server_id (server_1, server_2, etc.) as keys and inner maps as values. The inner map has auth parameter names as keys and auth parameter values as values.

So is there a simple way to handle this kind of configuration?

For now I work around the problem by changing the config to:

[
    {
        "id": "server_1",
        "auth": {
            "server_1_param_1": "value",
            "server_1_param_2": "value",
            "server_1_param_3": "value",
        }
    },
    {
        "id": "server_2",
        "auth": {
            "server_2_param_1": "value",
            "server_2_param_2": "value"
        }
    },
    ...
]

So there is no hurry making changes to the library, for me it is good enough the way it is now. It seems more natural to use two nested maps so it would be nice to have some way to use nested maps, but the workaround that I use is fine with me too.

[kamchatka-volcano]

Do you mean that currently FIGCONE_DICT supports maps where the value is a structure? Or does it require the value to always be a primitive, like std::string, int, bool, etc.?

It doesn't need to be a primitive, but it can only be read from a simple string value, by using the registered string conversion. Here's an example based on the "User defined types" README section:

struct Host {
    std::string ip;
    int port;
};

namespace figcone {
template<>
struct StringConverter<Host> {
    static std::optional<Host> fromString(const std::string& data)
    {
        auto delimPos = data.find(':');
        if (delimPos == std::string::npos)
            return {};
        auto host = Host{};
        host.ip = data.substr(0, delimPos);
        host.port = std::stoi(data.substr(delimPos + 1, data.size() - delimPos - 1));
        return host;
    }
};

struct Cfg{
    std::map<std::string, Host> testMap; 
}

{
    "testMap" : {
        "foo" : "127.0.0.1:8080",
        "bar" : "127.0.0.1:8000"
    }
}

So, with the Host structure as a map value it can't automatically parse the following config:

{
    "testMap" : {
        "foo" : {"ip": "127.0.0.1", "port":"8080"},
        "bar" : {"ip": "127.0.0.1", "port":"8080"},
    }
}

but I think it's possible to add support for it in the future.

As for your example

{
    "server_1": {
        "server_1_param_1": "value",
        "server_1_param_2": "value",
        "server_1_param_3": "value",
    },
    "server_2": {
        "server_2_param_1": "value",
        "server_2_param_2": "value",
    },
    ...
}

If "server_N" and "server_N_param_N" are arbitrary, and you cannot register a structure with expected field names, then no, there's no better way than using some combination of a node list and a dictionary, like the one you ended up with.

figcone's design helps to avoid using string-based maps when you have a defined schema configuration. At the same time, this shift away from using maps for storing configuration hurts flexibility when you need to support configurations with arbitrary key names, as in your example.

[MeanSquaredError]

If "server_N" and "server_N_param_N" are arbitrary, and you cannot register a structure with expected field names, then no, there's no better way than using some combination of a node list and a dictionary, like the one you ended up with.

figcone's design helps to avoid using string-based maps when you have a defined schema configuration. At the same time, this shift away from using maps for storing configuration hurts flexibility when you need to support configurations with arbitrary key names, as in your example.

My configuration is more complex than the example that I provided, because it has both a fixed part, which can be put into nested structures, and a dynamic part which is better suited for nested maps. The dynamic part, which in my case contains the auth data is handled by different plugins, so the main code, which parses the config, does not really know the parameter names of the auth data for each server id, so the best thing it can do is put the auth data into a std::unordered_map<string, string> and pass it to the corresponding plugin.

Anyway, thanks for the explanation. I am OK with the workaround that I use. I will leave the issue open in case you find the time to implement support for maps with string keys and structure values.