Scrubbing sensitive data from the spans (DD_APM_REPLACE_TAGS) does not support arrays

[mskarbe]

Describe what happened:

We have tried to use the feature of collecting the AWS Lambda functions payload following this documentation: https://docs.datadoghq.com/serverless/distributed_tracing/collect_lambda_payloads/ & https://docs.datadoghq.com/tracing/setup_overview/configure_data_security/?tab=environmentvariable#scrub-sensitive-data-from-your-spans and applying the environmental variables:

• DD_CAPTURE_LAMBDA_PAYLOAD set to true
• DD_APM_REPLACE_TAGS with valid array of { "name": "fieldName", "pattern": ".*", "repl": "REDACTED" } entries

Unfortunately, it appeared that DD Agent is not handling the obfuscation of arrays. I.e., if we set function.request.Records.0.body.arrayOfValuesToBeRedacted all the fields still come in to the Datadog dashboard with their original values instead of REDACTED. What solves it, is setting the names explicitly, one by one, like:
function.request.Records.0.body.arrayOfValuesToBeRedacted.0, function.request.Records.0.body.arrayOfValuesToBeRedacted.1, … , function.request.Records.0.body.arrayOfValuesToBeRedacted.n.

Nevertheless, this is not an option for cases where the number of entries in the list is varying and cannot be determined. It blocks us from using the given Datadog feature of payload collection.

Describe what you expected:

The current handling of tags replacement seems to be a bug. If we are excluding *.<array-name>, we would expect the entire field to be replaced and dropped; if we are specifying *.<array-name>.0 then only the first element in the given array.

Actually, the most feasible would be if the name field in the DD_APM_REPLACE_TAGS (i.e. replace_tag blocks) would support regex. This would allow to apply setup of payload obfuscation on the global level of our environment - based on the data model in the system itself - instead of on the function level, setting it for each lambda one by one.

Additional environment details (Operating System, Cloud provider, etc):

• AWS Lambda functions, Node.js 14 runtime
• datadog-lambda-js extension, v5.73.0
• dd-trace package, v2.3.1

[xrn]

Hey,

I have exactly the problem I would like to use it to standard PATCH request - https://www.rfc-editor.org/rfc/rfc6902

 [
     { "op": "replace", "path": "/credencials/password", "value": "myPassword" },
     { "op": "remove", "path": "/age" }
]

@paulcacheux @julien-lebot