How to serialize User-Defined Structures in the libnop library?

[vajos]

The libnop github about user defined structures: https://github.com/google/libnop/blob/master/docs/getting-started.md#user-defined-structures

"The easiest and most flexible way annotate a user-defined type is by using the macro NOP_STRUCTURE(type, ... / members /)."

My Code with a simple std::map:

My struct:

struct EVHand
{
    double added_values = 0;
    uint64_t count_hands = 0;
    NOP_STRUCTURE(EVHand, added_values, count_hands);
};

std::string GetTextFile(const std::string& file_path)
{
    const std::ifstream myfile(file_path, std::ifstream::binary | std::ifstream::in);
    if (myfile.is_open())
    {
        std::stringstream buffer;
        buffer << myfile.rdbuf();

        return buffer.str();
    }

    return "";
}

namespace {

    // Sends fatal errors to std::cerr.
    auto Die() { return nop::Die(std::cerr); }

}  // anonymous namespace

Serialize Data

template <typename T>
void SerializeDatatype(const T& value) {
    // Create a serializer around a std::stringstream.
    using Writer = nop::StreamWriter<std::stringstream>;
    nop::Serializer<Writer> serializer;

    // Write some data types to the stream.
    serializer.Write(value) || Die();

    const std::string data = serializer.writer().stream().str();
    std::cout << "Wrote " << data.size() << " bytes." << std::endl;
    //Save data

    std::string path = std::string("C:\\Log\\") + "test.txt";
    try {
        std::ofstream ofs;
        ofs.open(path, std::ofstream::binary | std::ofstream::out);

        ofs << data;
        ofs.close();
    }
    catch (std::exception & e)
    {
        std::cout << e.what() << std::endl;
    }

}

Read Serialized Data

template<class T>
T ReadSerializedDatatype()
{

  //1. Read whole Text File
    const std::string file = GetTextFile("C:\\Log\\test.txt");
    // Create a deserializer around a std::stringstream with the serialized data.
    using Reader = nop::StreamReader<std::stringstream>;
    nop::Deserializer<Reader> deserializer{ file };

    // Read some data types from the stream.

    T map_value;

    deserializer.Read(&map_value) || Die();

    return map_value;
}

Main

int main(int, char**) {

    std::map<double,EVHand> map; 

    double amount = 2.5;
    auto amount_key_not_found = map.find(amount) == map.end();

    if (amount_key_not_found)
    {
        map.insert(std::make_pair(amount, EVHand()));
    }

    double amount_won = 4;

    EVHand test;
    test.added_values = amount_won + amount;
    test.count_hands = map.at(amount).count_hands + 1;
    map[amount] = test;

    std::cout << "\n";

    std::cout << "Added values: " << map.at(amount).added_values;
    std::cout << "\n";
    std::cout << "counted hands: " << map.at(amount).count_hands;

    std::cout << "\n";

    SerializeDatatype(map);

    map = ReadSerializedDatatype< std::map<double, EVHand>>();

    for (auto const& [key, val] : map)
    {
        std::cout <<  "Key: " <<key << "\n";
        std::cout << "Counted hands: " << val.count_hands << "\n";
        std::cout << "Value: " << val.added_values << "\n";
    }

    return 0;
} 

Results:

Added values: 6.5
counted hands: 1
Wrote 13 bytes.
Key: 2.5
Counted hands: 0
Value: 0

Changing the std::map value to INT produces correct serialization. So there has to be something wrong with how I NOP_STRUCT my struct?

Im not sure if questions like that are allowed here but I saw the Help wanted label so I guess its ok?

[eieio]

It's perfectly fine to ask questions here. :-)

First off, your structure definition looks correct. I don't think this is the problem.

I'm not sure how you got a payload of 13 bytes with double as a key and having a double in your element structure. A double always costs 9 bytes in serialized format, and since you have one for the map key and one in the element, the serializer should produce more than 18 bytes for this input, guaranteed.

Second, when I make a reproduction case that does not include file IO things seem to work fine. Take a look at this example: https://godbolt.org/z/6eYiXd

I suspect your data is getting truncated. You should make sure your file IO is correct in both directions.

The actual serialized data bytes should be: BB 01 89 00 00 00 00 00 00 04 40 B9 02 89 00 00 00 00 00 00 1A 40 01.

If you truncate to the first 14 bytes and then zero filled the rest it would look like this: BB 01 89 00 00 00 00 00 00 04 40 B9 02 89 00 00 00 00 00 00 00 00 00.

I'm not quite sure why this is happening, but it does seem to explain the result you got. Your file IO is probably the culprit here.

To make things simpler, you can build a nop::Serializer<nop::StreamWriter<std::ofstream>> and a nop::Deserializer<nop::StreamReader<std::ifstream>> directly and avoid the intermediate std::stringstream and std::string buffer. This might address the file IO issues you are having.

Lastly, you should avoid using double for the key of std::map. A small change in value due to limited accuracy/rounding during double precision floating point computations would result in the wrong element being indexed or std::map::find() failing to locate a previously inserted element. If you need to represent money values you should use a decimal type, which is usually based on an integer.

Best, Corey