Bad Request when sending logs to datadog

kaisermario commented 2 years ago

Describe what happened: I receive 400 http errors when using output plugin fluent-plugin-datadog. Unfortunately I can not see any log which give me an hint for the cause.

Describe what you expected:

Steps to reproduce the issue:

Enclosed you can find my configuration:

@type stdout
</filter>

<filter **>
@type concat
@log_level debug
key log
multiline_start_regexp /\"timestamp\":\"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z\"/
multiline_end_regexp /\"endOfBatch\":.*}/
separator ""
</filter>

<filter **>
@type parser
key_name log
reserve_data false
reserve_time true
<parse>
@type multi_format
<pattern>
format json
keep_time_key true
</pattern>
<pattern>
format none
</pattern>
</parse>
</filter>

<filter **>
@type record_transformer
<record>
environment "#{ENV['STAGE']}-#{ENV['REGION']}"
</record>
</filter>

<match **>
@type datadog
host "http-intake.logs.datadoghq.eu"
api_key "#{ENV['DD_API_KEY']}"
dd_tags "#{ENV['DD_TAGS']}"
dd_source "#{ENV['DD_SOURCE']}"
dd_service "#{ENV['SERVICENAME']}"
</match>

one example log:

{
   "instant":{
      "epochSecond":1633338876,
      "nanoOfSecond":111227000
   },
   "thread":"RecordProcessor-0000",
   "level":"DEBUG",
   "loggerName":"com.amazonaws.services.kinesis.clientlibrary.lib.worker.RecordProcessorCheckpointer",
   "message":"Checkpointing shardId-000000000000, token 2e24d703-b553-412f-99d7-abbcc99bdd26 at largest permitted value {SequenceNumber: 49622652083619008019359084295624474801284082641473110018,SubsequenceNumber: 0}",
   "endOfBatch":false,
   "loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog",
   "contextMap":{

   },
   "threadId":79,
   "threadPriority":5
}

Additional environment details (Operating System, Cloud provider, etc):

docker file:

FROM fluent/fluentd:v1.14

USER root

RUN apk add --no-cache --update --virtual .build-deps \

sudo build-base ruby-dev \

&& sudo gem install fluent-plugin-concat \

&& sudo gem install fluent-plugin-sumologic_output \

&& sudo gem install fluent-plugin-datadog \

&& sudo gem install fluent-plugin-multi-format-parser \

&& sudo gem sources --clear-all \

&& apk del .build-deps \

&& rm -rf /home/fluent/.gem/ruby/2.5.0/cache/*.gem

COPY fluent.conf /fluentd/etc/custom/

COPY fluent_dd.conf /fluentd/etc/custom/

COPY entrypoint.sh /bin/

RUN chmod +x /bin/entrypoint.sh

USER fluent

I would be happy to get an hint. I am little bit frustated meanwhile.

remeh commented 2 years ago

Hello @kaisermario, Sorry for the late reply. Are you still receiving 400s using the plugin?

remeh commented 2 years ago

Closing because of no activity. Please re-open if that makes sense 👍

DataDog / fluent-plugin-datadog

Bad Request when sending logs to datadog #49