microsoft / bond

Bond is a cross-platform framework for working with schematized data. It supports cross-language de/serialization and powerful generic mechanisms for efficiently manipulating data. Bond is broadly used at Microsoft in high scale services.
MIT License
2.61k stars 321 forks source link

Correct way to read list of mapped values in C# #1163

Closed dend closed 2 years ago

dend commented 2 years ago

I have a list (BT_LIST) of mapped values (BT_MAP) that seems to contain string values (BT_STRING). I am reading the container data as such (including BT_MAP):

private void ReadContainer(CompactBinaryReader<Bond.IO.Unsafe.InputBuffer> reader)
{
    reader.ReadContainerBegin(out int containerCounter, out BondDataType containerDataType);

    for (int i = 0; i < containerCounter; i++)
    {
        this.DecideOnDataType(reader, containerDataType);
    }

    reader.ReadContainerEnd();
}

It seems, however, that while the function above works for lists and sets, it doesn't work for BT_MAP because I am not fully reading the header data (key type, value type, count). Is there an appropriate routine for this @chwarr?

dend commented 2 years ago

I updated the code a bit - I can read the value data type as well, but still unable to read actual values:

private void ReadContainer(CompactBinaryReader<Bond.IO.Unsafe.InputBuffer> reader, bool isMap = false)
{
    string marker = "Mapped value type: ";
    int containerCounter;
    BondDataType containerDataType = BondDataType.BT_UNAVAILABLE;
    BondDataType valueDataType = BondDataType.BT_UNAVAILABLE;

    if (!isMap)
    {
        reader.ReadContainerBegin(out containerCounter, out containerDataType);
    }
    else
    {
        reader.ReadContainerBegin(out containerCounter, out containerDataType, out valueDataType);
    }

    for (int i = 0; i < containerCounter; i++)
    {
        this.DecideOnDataType(reader, containerDataType);
    }

    reader.ReadContainerEnd();
}

How do I read the full mapped value here?

chwarr commented 2 years ago

Maps are serialized in Compact Binary as contiguous key/value pairs. Your DecideOnDataType doesn't take a valueDataType parameter, so my guess is that it doesn't know how to read the value in each map entry's key/value pair.

The .expressions files under cs/test/expressions show the Linq expressions that each transform generates for the schema in the same directory. Those could be useful to consult to figure out how Bond deserializes a given construct. For example, take a look at the code generated to deserialize 60: map<int32, double> _map. The core loop reads an int32 then a double count times where count is the size that ReadContainerBegin reported.

.Block(
    System.Int32 $Example_result._map_key,
    System.Double $Example_result._map_value) {
    .Default(System.Void);
    .Loop  {
        .If ($count-- > 0) {
            .Block() {
                $Example_result._map_key = .Call $reader.ReadInt32();
                .Default(System.Void);
                $Example_result._map_value = .Call $reader.ReadDouble();
                ($Example_result._map).Item[$Example_result._map_key] = $Example_result._map_value
            }
        } .Else {
            .Break end { }
        }
    }
    .LabelTarget end:

There's a lot more code there, but it's also handling numeric promotions for both the key and the value as well as skipping unknown fields.

dend commented 2 years ago

Aha - I think I see that the right approach here is that once I read the map key, I also need to read the value sequentially. Here is my implementation for it: https://github.com/OpenSpartan/bond-reader/blob/e5a6b454721e81aa3766a8e0eca55aa0e6b73795/src/BondReader/BondProcessor.cs#L101-L109

It seems, at least on the surface, to work - because the value is a struct, it goes into reading the container again and seems to yield the right values back.

Thank you again for helping look into this @chwarr!