DataDog / kong-plugin-ddtrace

Datadog APM Plugin for Kong Gateway
Apache License 2.0
15 stars 7 forks source link

Invalid trace sent when no routes matched #5

Closed danielpoonwj closed 10 months ago

danielpoonwj commented 1 year ago

Hi,

I'm seeing these logs from the Datadog Agent when Kong encounters requests that don't match any routes. Maybe some data is missing in those cases and it's resulting in a broken tracing request.

2023-07-17 14:16:20 UTC | TRACE | ERROR | (run.go:284 in Errorf) | Cannot decode v0.4 traces payload: msgp: attempted to decode type "array" with method for "map" at 0/1/Meta

I'm not sure if the expected behaviour is to send traces in these scenarios or not, but it seems that tracing requests are still being sent and they're invalid/cannot be decoded by the Agent.

dgoffredo commented 1 year ago

I'm only speculating, but maybe an empty table is "without hole," and so an empty meta property (tags) gets encoded as an array instead of as a map: https://github.com/DataDog/kong-plugin-ddtrace/blob/b9fbbb87e1d03f2ed7183937e2008698abf3e805/kong/plugins/ddtrace/msgpack_encode.lua#L232

Ah, I misunderstood the code. Still, the issue could be related.

danielpoonwj commented 1 year ago

I think this is somehow related. This line https://github.com/DataDog/kong-plugin-ddtrace/blob/b7f88ed6c7b92e4af09c20b52c0eb5b1c0151864/kong/plugins/ddtrace/sampler.lua#L10

should actually be

cjson.decode_array_with_array_mt(true)

https://github.com/Kong/kong-cjson/blob/master/README.md#decode_array_with_array_mt

I found this out due to errors like this on a lower priority plugin where it tries to call the boolean variable instead of a function. Not directly related but indicative of a bug introduced when I added this plugin. Side note, I'm not sure how this plugin can modify functions of cjson affecting other plugins. Maybe the required modules are cached and mutable, and this plugin is higher priority than the one where I encountered that error.

attempt to call field 'decode_array_with_array_mt' (a boolean value)

I think it's related because it directly affects the serialization/deserialization of arrays and maps

danielpoonwj commented 1 year ago

Hi @cgilmour thank you for the release v0.1.1, that resolved the issue.

Could you take a look at this syntax too even if it's unrelated to this issue. More information in my earlier comment before this. https://github.com/DataDog/kong-plugin-ddtrace/blob/b7f88ed6c7b92e4af09c20b52c0eb5b1c0151864/kong/plugins/ddtrace/sampler.lua#L10

It's causing issues with other plugins when using the function because the function is overridden

cjson.decode_array_with_array_mt(true)
danielpoonwj commented 1 year ago

I can confirm that changing

cjson.decode_array_with_array_mt = true 

to

cjson.decode_array_with_array_mt(true)

resolved this error on from my other plugin

attempt to call field 'decode_array_with_array_mt' (a boolean value)
dmehala commented 10 months ago

The issue have been fixed in #10 and is available in v0.1.1. Thank you for reporting the issue.