Open deeso opened 10 months ago
That looks correct, happy to accept a PR.
If you wish to continue using the existing table, you will have to manually drop or rename the column from your table via Spark or API since it's a breaking schema change (if you're testing you can just recreate it).
There is a bug in the the cloudflare http event schema. The schema defines the
cloudflare.http_event.bot.tag
as a string here, but the actual value is an array of strings, see: https://github.com/matanolabs/matano/blob/b9975f5e92a3c9aedca2e8879bb4b81f6861eb97/data/managed/log_sources/cloudflare/tables/http_request.yml#L60When the VRL parses the log, the result is either null or an array of strings at this location: https://github.com/matanolabs/matano/blob/b9975f5e92a3c9aedca2e8879bb4b81f6861eb97/data/managed/log_sources/cloudflare/tables/http_request.yml#L457
This causes any JSON log line containing a
BotTags
array to fail and be sidelined by the transform. The error creates the following error message in the CloudWatch logs for the Data Transformer lambda:To fix this issue, this block snippet needs to be converted from:
To: