[[inputs.kafka_consumer]] ==> [[outputs.datadog]]

[viveklanjekar]

I am trying to get data from Kafka consumer to be sent to datalog but while doing so, I am getting below error in telegraf logs.

2018-03-16T08:04:20Z E! Error writing to output [datadog]: error POSTing metrics, Post https://app.datadoghq.com/api/v1/series?api_key=****: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

PS: I am writing in the above-mentioned kafka consumer with telegraf running on other hosts and the data type is infux

below is the format which is stored in kafka consumers

disk,path=/hostfs/app,device=mapper/vg_app-lv_app,fstype=ext4,mode=rw,host=jira01.sg.aws.grabpay.com total=32740589568i,free=31379046400i,used=3170304i,used_percent=0.01010223092238067,inodes_total=2023680i,inodes_free=2023648i,inodes_used=32i 1521187170000000000

docker_container_blkio,container_image=zabbix/zabbix-agent,org.label-schema.usage=https://www.zabbix.com/documentation/3.4/manual/installation/containers,org.label-schema.docker.cmd=docker\ run\ --name\ zabbix-agent\ --link\ zabbix-server:zabbix-server\ -p\ 10050:10050\ -d\ zabbix-agent:ubuntu-3.4.7,device=202:6144,org.label-schema.version=3.4.7,org.label-schema.vendor=Zabbix\ LLC,container_version=ubuntu-3.4-latest,host=dev-gpdm01.sg.aws.grabpay.com,org.label-schema.vcs-url=svn://svn.zabbix.com/tags/3.4.7/,org.label-schema.build-date=2018-02-23T04:55:47Z,engine_host=dev-gpdm01.sg.aws.grabpay.com,org.label-schema.schema-version=1.0,org.label-schema.description=Zabbix\ agent\ is\ deployed\ on\ a\ monitoring\ target\ to\ actively\ monitor\ local\ resources\ and\ applications,org.label-schema.vcs-ref=3.4.7,org.label-schema.license=GPL\ 2.0,container_name=zabbix-agent,org.label-schema.name=zabbix-agent-ubuntu,org.label-schema.url=https://zabbix.com/,maintainer=Alexey\ Pustovalov\ alexey.pustovalov@zabbix.com io_service_bytes_recursive_write=417792i,io_service_bytes_recursive_sync=0i,io_service_bytes_recursive_async=10657792i,io_service_bytes_recursive_total=10657792i,io_serviced_recursive_async=281i,io_service_bytes_recursive_read=10240000i,io_serviced_recursive_read=247i,io_serviced_recursive_write=34i,io_serviced_recursive_sync=0i,io_serviced_recursive_total=281i,container_id="e01572b9235f1e239c172f8bd1cfecb91f1a46f8cf46b8361804d8d4f84546c2" 1521187162000000000

disk,path=/hostfs/var/lib/docker/overlay2/9576a5812ad2e4395d689400a68491f413d02a30bac47c28141e8b2536ea5297/merged,device=overlay,fstype=overlay,mode=rw,host=jenkins01.sg.aws.corp.grabpay.com inodes_total=32705280i,inodes_free=29298484i,inodes_used=3406796i,total=529511669760i,free=391376400384i,used=116591050752i,used_percent=22.952464865860993 1521187170000000000

[danielnelson]

This looks a network issues between you and Datadog, it might be helpful to try sending data with curl:

https://docs.datadoghq.com/api/?lang=bash#post-time-series-points

[viveklanjekar]

hi @danielnelson i checked connectivity is not the issue, as the machine is able to connect to datalog. if i remove the timeout parameter from outputs.datalog, i get following error.

2018-03-18T18:47:00Z W! Skipping a scheduled flush because there is already a flush ongoing. 2018-03-18T18:47:00Z E! Error writing to output [datadog]: received bad status code, 408

[danielnelson]

Can you show the output of this curl command they show in their api documentation, it might be helpful to run it with -v as well:

currenttime=$(date +%s)

curl -v -X POST -H "Content-type: application/json" \
-d "{ \"series\" :
         [{\"metric\":\"test.metric\",
          \"points\":[[$currenttime, 20]],
          \"host\":\"test.example.com\",
          \"tags\":[\"environment:test\"]}
        ]
}" \
'https://app.datadoghq.com/api/v1/series?api_key=9775a026f1ca7d1c6c5af9d94d9595a4'

[viveklanjekar]

hi, @danielnelson figured out the problem,

Datadog API times out for metric_batch_size of 1000 so now set to 500 metric_batch_size = 500

[viveklanjekar]

Also please Hash out your api_key if its used anywhere , thanks for your help

[danielnelson]

Thanks for the update, glad it is working. Luckily that api key is just the example one from datadog ;)