truncated dynamic value while querying with `v1.0.0-preview-4`

[weikanglim]

Describe the bug I am getting an error while using query.ToStructs. The error is:

• Error occurred while trying to unmarshal Dynamic into a map: json: string unexpected end of JSON input

This started happening on v1.0.0-preview-4. This same code is working with v1.0.0-preview-3, I've switched back in the meantime.

As a brief debugging step, I dumped the string that is being processed in dynamic.go and confirmed that the JSON-string is truncated (malformed), with a length of exactly 1275.

To Reproduce Pseudo-code (without the database connection information):

type Event struct {
    Props map[string]interface{} `kusto:"event_properties"`
}

kqlQuery := kql.New(`datatable (event_properties: dynamic) [
    dynamic({"first_field":"[\"elem1\",\"elem2\"]","second_field":"0000000000000000"})
]`)

res, err := client.Query(ctx, "<database>", kqlQuery)
if err != nil {
    return fmt.Errorf("querying: %w", err)
}

// print out what Row looks like
rows := res.Tables()[0].Rows()
for _, row := range rows {
    props, err := row.DynamicByName("event_properties")
    if err != nil {
        return fmt.Errorf("getting event properties: %w", err)
    }
    fmt.Println("ROW")
    fmt.Println(row.String())
    fmt.Println("ROW")

    fmt.Println("PROPS")
    fmt.Println(string(props))
    fmt.Println("PROPS")
}

// test the failure condition
structs, err := query.ToStructs[Event](res)
if err != nil {
    return fmt.Errorf("to structs: %w", err)
}

Output:

ROW
"{""first_field"":""[\""elem1\"",\""elem2\""]"",""second_field"":""00000000000000"

ROW
PROPS
{"first_field":"[\"elem1\",\"elem2\"]","second_field":"00000000000000
PROPS
{"first_field":"[\"elem1\",\"elem2\"]","second_field":"00000000000000
to structs: Op(Op(8)): Kind(Kind(11)): column event_properties could not store in struct.EventProps: Error occurred while trying to unmarshal Dynamic into a map: json: string unexpected end of JSON input
exit status 1

Expected behavior

toStructs shouldn't fail handling valid, but long (1275+ long) dynamic property bags.

[AsafMah]

Thank you, looking into this.

If you could provide us with a sample to what the field looks like it could be very helpful.

I tried it with a longer value (2000 characters), and hadn't gotten this error.

Can you provide also the schema of the table?

[weikanglim]

@AsafMah

I managed to take a second look, and reduced the minimal repro to a KQL query that looks like:

datatable (event_properties: dynamic) [
    dynamic({"first_field":"[\"elem1\",\"elem2\"]","second_field":"0000000000000000"})
]

I updated the repro steps above with the pseudo-code and the output I observed. It seems that when the property bag has a JSON-array element, client.Query may not be handling this correctly.

The decoded JSON in row.Values is truncated:

{"first_field":"[\"elem1\",\"elem2\"]","second_field":"00000000000000

[AsafMah]

Thank you for the detailed research!

It seems that the culprit is the new json library we used, while it did gave us some performance benefits, it seems that it doesn't handle edge cases well.

We will release a version soon which is back to the standard json library.

[AsafMah]

Hey, try preview-5 please

[AsafMah]

It's been a week since the new release, since the version has a fix I'm going to close this issue.

Feel free to comment or re-open as needed.

[weikanglim]

@AsafMah thanks for the fix. I can confirm the behavior is fixed with preview-5.