search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.6k stars 5.57k forks source link

Telegraf not using AWS endpoints for timestream #10265

Closed nathleeuk closed 2 years ago

nathleeuk commented 2 years ago

Relevent telegraf.conf

[[outputs.timestream]]
  ## Amazon Region
  region = "eu-west-1"
  ## Amazon Credentials
  access_key = "REVOKED"
  secret_key = "REVOKED"
  #token = ""
  #role_arn = ""
  #web_identity_token_file = ""
  #role_session_name = ""
  #profile = ""
  #shared_credential_file = ""
  ## Endpoint to make request against, the correct endpoint is automatically
  ## determined and this option should only be set if you wish to override the
  ## default.
  ##   ex: endpoint_url = "http://localhost:8000"
  #endpoint_url = "https://ingest-cell1.timestream.eu-west-1.amazonaws.com"
  #endpoint_url = "ingest-cell1.timestream.eu-west-1.amazonaws.com"
  ## Timestream database where the metrics will be inserted.
  ## The database must exist prior to starting Telegraf.
  database_name = "Celerity_Test_Timestream"
  ## Specifies if the plugin should describe the Timestream database upon starting
  describe_database_on_start = false
  ## The mapping mode specifies how Telegraf records are represented in Timestream.
  mapping_mode = "multi-table"
  ## Specifies if the plugin should create the table, if the table do not exist.
  create_table_if_not_exists = true
  ## Only valid and required if create_table_if_not_exists = true
  create_table_magnetic_store_retention_period_in_days = 365
  ## Only valid and required if create_table_if_not_exists = true
  create_table_memory_store_retention_period_in_hours = 24

System info

Telegraf 1.20.4, Amazon Linux 2

Docker

not using docker

Steps to reproduce

  1. create timestream database in AWS
  2. use standard configuration to send data
  3. this works fine but by default telegraf will send the data over the internet
  4. create an AWS endpoint for timestream to allow data to stay private (does not go over the internet)
  5. telegraf will not use the AWS endpoint by default and there is no way of telling telegraf to use private endpoint

Expected behavior

somewhere in the telegraf config to override the timestream url with the vpc endpoint

Actual behavior

2021-12-13T17:27:45Z E! [outputs.timestream] Failed to write to Timestream database 'Celerity_Test_Timestream' table 'processes'. Skipping metric! Error: 'operation error Timestream Write: WriteRecords, operation error Timestream Write: DescribeEndpoints, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , request send failed, Post "https://ingest.timestream.eu-west-1.amazonaws.com/": dial tcp 54.76.10.207:443: i/o timeout'

Additional info

Spoke to Josh Powers original who understands the issue and has done some initial testing

powersj commented 2 years ago

@nirmeshk is accessing Timestream via a VPC endpoint something that the golibary knows how to do? Or is there something Telegraf needs to learn to do?

powersj commented 2 years ago

@nirmeshk wanted to ping you again on this one Timestream + VPC usage. Is there something Telegraf needs to learn to do or something the AWS library knows how to work with?

powersj commented 2 years ago

@sjwang90 who else from the timestream team can we ask?

sjwang90 commented 2 years ago

@piotrwest @sudiptodas @cnov @NeoZHL @zhihaoli - would any of you be able to answer the following:

is accessing Timestream via a VPC endpoint something that the golibary knows how to do? Or is there something Telegraf needs to learn to do?

sudiptodas commented 2 years ago

Hi,

Thank you for reporting this issue. We're actively working on addressing this issue. We have an upcoming PR which would address this issue. We apologize for the inconvenience.

Thank you Amazon Timestream team.

nathleeuk commented 2 years ago

Thanks team,

Do you have any rough time scales for the PR?

Cheers,

Nathan

From: Sudipto Das @.> Reply to: influxdata/telegraf @.> Date: Monday, 14 February 2022 at 21:52 To: influxdata/telegraf @.> Cc: Nathan Lee @.>, Author @.***> Subject: Re: [influxdata/telegraf] Telegraf not using AWS endpoints for timestream (Issue #10265)

CAUTION: This email is from an external source outside of Celerity. Do not click links, open attachments, reply or forward internally unless you know and trust the sender.

Hi,

Thank you for reporting this issue. We're actively working on addressing this issue. We have an upcoming PR which would address this issue. We apologize for the inconvenience.

Thank you Amazon Timestream team.

— Reply to this email directly, view it on GitHubhttps://github.com/influxdata/telegraf/issues/10265#issuecomment-1039605501, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASBPTJLP2RSSFSQZVDMQA53U3F2SDANCNFSM5KAHV2AQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.Message ID: @.***>

Nathan Lee Senior DevOps Engineer 07455 009021 | Privacy Policy www.celerity-uk.com | @.***

​The content of this email is confidential and intended for the recipient specified in the message only and is subject to copyright. ​​If you received this message in error, please reply to this message and follow with its deletion. It is strictly forbidden to share any part of this message with any third party, without written consent of the sender. ​​Please note that e-mails are subject to viruses, data corruption, delay, interception and unauthorised amendment, and that the sender does not accept liability for any damages that may be incurred as a result of communication by email. ​ Celerity Limited is a company registered in England and Wales, Registered number: 4475496. Registered office: Celerity Limited, 11 St George’s Court, Kirkham, Preston, PR4 2EF. Please click here to view our Privacy Policy.

nathleeuk commented 2 years ago

Hi Team,

Did you manage to check the timescales for the PR?

Cheers,

Nathan Lee Senior DevOps Engineer 07455 009021 | Privacy Policy www.celerity-uk.com | @.***

​The content of this email is confidential and intended for the recipient specified in the message only and is subject to copyright. ​​If you received this message in error, please reply to this message and follow with its deletion. It is strictly forbidden to share any part of this message with any third party, without written consent of the sender. ​​Please note that e-mails are subject to viruses, data corruption, delay, interception and unauthorised amendment, and that the sender does not accept liability for any damages that may be incurred as a result of communication by email. ​ Celerity Limited is a company registered in England and Wales, Registered number: 4475496. Registered office: Celerity Limited, 11 St George’s Court, Kirkham, Preston, PR4 2EF. Please click here to view our Privacy Policy. From: Nathan Lee @.> Date: Tuesday, 15 February 2022 at 14:33 To: influxdata/telegraf @.>, influxdata/telegraf @.> Cc: Author @.> Subject: Re: [influxdata/telegraf] Telegraf not using AWS endpoints for timestream (Issue #10265) Thanks team,

Do you have any rough time scales for the PR?

Cheers,

Nathan

From: Sudipto Das @.> Reply to: influxdata/telegraf @.> Date: Monday, 14 February 2022 at 21:52 To: influxdata/telegraf @.> Cc: Nathan Lee @.>, Author @.***> Subject: Re: [influxdata/telegraf] Telegraf not using AWS endpoints for timestream (Issue #10265)

CAUTION: This email is from an external source outside of Celerity. Do not click links, open attachments, reply or forward internally unless you know and trust the sender.

Hi,

Thank you for reporting this issue. We're actively working on addressing this issue. We have an upcoming PR which would address this issue. We apologize for the inconvenience.

Thank you Amazon Timestream team.

— Reply to this email directly, view it on GitHubhttps://github.com/influxdata/telegraf/issues/10265#issuecomment-1039605501, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASBPTJLP2RSSFSQZVDMQA53U3F2SDANCNFSM5KAHV2AQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.Message ID: @.***>

sudiptodas commented 2 years ago

Hi Nathan,

Thank you for your patience. As we're working through it, we're also trying to project a more accurate date when it will be ready. If you're blocked, could you drop a note to sudiptod -- AT -- amazon.com and we may be able to unblock you.

Thanks Amazon Timestream team

sudiptodas commented 2 years ago

PR for the fix is out: https://github.com/influxdata/telegraf/pull/10841