Endpoint fix from pull request 27024 not working in 7.14.0 docker image

[bkennedy-mx]

This is in regards to: https://github.com/elastic/beats/pull/27024

We were asked to open an issue, as our issue in 7.12.0, which was the catalyst for this pull request, is still occurring in 7.14.0. The error with both the 7.12.0 and 7.14.0 docker images is (with sensitive portions removed):

2021-07-20T20:57:06.177Z ERROR [aws-cloudwatch] awscloudwatch/input.go:154 getLogEventsFromCloudWatch failed: RequestError: send request failed caused by: Post "https://cloudwatchlogs.XXXXXXXXXXX.sc2s.sgov.gov/": dial tcp: lookup cloudwatchlogs.XXXXXXXX.sc2s.sgov.gov on 10.X.X.X:53: no such host

Debug level logging does not provide any addition messages. The above was from 7.12.0 and the 7.14.0 errors are identical. Original logs are attached to Support Case 00756862, and I am attaching them here as well. It will take about a week to get a new copy of the logs cleansed and pulled down to unclass space, and the new logs have no difference in the messages shown.

I've attached the helm template for our config. I'll pull and cleanse a copy of the rendered configmap when I do the transfer of the logs.

rds-filebeat_2021-07-20_redacted.log rds-filebeat-deployment-module-configmap.yml.txt

[asazallesmilner]

@kaiyan-sheng This is the issue you requested. Thanks!

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[kaiyan-sheng]

@bkennedy-mx Thanks for creating the issue. Could you actually post the error log from 7.14 please?

[bkennedy-mx]

I was able to get into SecOps yesterday, but we have an issue with our kubernetes environment that prevented me from getting them. If that issue gets resolved, I'll be able to download them, cleanse them and start the data transfer tomorrow, but it could be up to a week before the transfer gets done. I'll get them as soon as I can. To be clear, the errors in the logs are identical to those in 7.12.0.

[asazallesmilner]

rds-filebeat_logs_2021-08-16_cleansed.txt

Here is the cleaned logs from the secure environment in debug mode showing it using the wrong URL.

[legoguy1000]

What do u have set as the endpoint value??

          {{- if $.Values.pipeline.AWS_ENDPOINT }}
          endpoint: "{{ $.Values.pipeline.AWS_ENDPOINT }}"
          {{- end }}

[asazallesmilner]

We provide the sc2s.sgov.gov That works and changes the suffix just fine. The problem is that it ALSO changes the beginning of the url to cloudwatchlogs instead of just logs like it should be when it does it.

Note: we did test this both from EC2 AND from the Docker container in kubernetes.

so it looks like the change from cwConfig := awscommon.EnrichAWSConfigWithEndpoint(in.config.AwsConfig.Endpoint, "cloudwatchlogs", in.config.RegionName, in.awsConfig) to cwConfig := awscommon.EnrichAWSConfigWithEndpoint(in.config.AwsConfig.Endpoint, "logs", in.config.RegionName, in.awsConfig)

is not being applied for some reason.

[legoguy1000]

Ya the endpoint is good. Definitely strange.

[kaiyan-sheng]

@asazallesmilner Hi! Are you using 7.14.0 for testing? Sorry I just double checked and this change is merged into 7.14 branch but missed 7.14.0 build. So it will be shipped in 7.14.1 and 7.15.0 release. 7.14.1 should be coming fairly soon (in days...).

[asazallesmilner]

Yes, we are using 7.14.0. We will look out for 7.14.1.

From: kaiyan-sheng @.> Sent: Tuesday, August 24, 2021 12:03 PM To: elastic/beats @.> Cc: Zalles-Milner, Asa @.>; Mention @.> Subject: Re: [elastic/beats] Endpoint fix from pull request 27024 not working in 7.14.0 docker image (#27262)

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

@asazallesmilnerhttps://urldefense.us/v2/url?u=https-3A__github.com_asazallesmilner&d=DwMCaQ&c=qqkkpu_zF8amsdnRZA_Et2-uNBtAipSVjV2iUVt238g&r=2FY5ZIPhxlzhT5oRG2zM_pT1HCmvyu01qaznhqAvB1KbjKkq3czivLM8QxnO9n4X&m=imXcx1zWWedaepzSdKCDRhSw4YlfVtyPvFpXgf8-2VM&s=MZPSr95DLilZ9vqVKoYoiMV9E1sbEubf0CCvL4ZyYkE&e= Hi! Are you using 7.14.0 for testing? Sorry I just double checked and this changehttps://urldefense.us/v2/url?u=https-3A__github.com_elastic_beats_pull_27007&d=DwMCaQ&c=qqkkpu_zF8amsdnRZA_Et2-uNBtAipSVjV2iUVt238g&r=2FY5ZIPhxlzhT5oRG2zM_pT1HCmvyu01qaznhqAvB1KbjKkq3czivLM8QxnO9n4X&m=imXcx1zWWedaepzSdKCDRhSw4YlfVtyPvFpXgf8-2VM&s=nUUPcBjtVxQcpiiaLNbugWxTYF1z24lK2nF9-bT_0fE&e= is merged into 7.14 branch but missed 7.14.0 build. So it will be shipped in 7.14.1 and 7.15.0 release. 7.14.1 should be coming fairly soon (in days...).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.us/v2/url?u=https-3A__github.com_elastic_beats_issues_27262-23issuecomment-2D904858876&d=DwMCaQ&c=qqkkpu_zF8amsdnRZA_Et2-uNBtAipSVjV2iUVt238g&r=2FY5ZIPhxlzhT5oRG2zM_pT1HCmvyu01qaznhqAvB1KbjKkq3czivLM8QxnO9n4X&m=imXcx1zWWedaepzSdKCDRhSw4YlfVtyPvFpXgf8-2VM&s=EW-zrLWmZTpTkKmnL9RXm5agfYxSmna3dCaruCU6nU8&e=, or unsubscribehttps://urldefense.us/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALZBDAVOQEKZXMIEOSMU7E3T6PNEZANCNFSM5BUHR5VQ&d=DwMCaQ&c=qqkkpu_zF8amsdnRZA_Et2-uNBtAipSVjV2iUVt238g&r=2FY5ZIPhxlzhT5oRG2zM_pT1HCmvyu01qaznhqAvB1KbjKkq3czivLM8QxnO9n4X&m=imXcx1zWWedaepzSdKCDRhSw4YlfVtyPvFpXgf8-2VM&s=m4Lpd5-fET26MiW_3u7tXU9sJ2yb0XB6NHoXrSoss2o&e=. Triage notifications on the go with GitHub Mobile for iOShttps://urldefense.us/v2/url?u=https-3A__apps.apple.com_app_apple-2Dstore_id1477376905-3Fct-3Dnotification-2Demail-26mt-3D8-26pt-3D524675&d=DwMCaQ&c=qqkkpu_zF8amsdnRZA_Et2-uNBtAipSVjV2iUVt238g&r=2FY5ZIPhxlzhT5oRG2zM_pT1HCmvyu01qaznhqAvB1KbjKkq3czivLM8QxnO9n4X&m=imXcx1zWWedaepzSdKCDRhSw4YlfVtyPvFpXgf8-2VM&s=KBw9qpCF925kLdXjdStkp9Wc0M5PTObeJs2g9mWtprY&e= or Androidhttps://urldefense.us/v2/url?u=https-3A__play.google.com_store_apps_details-3Fid-3Dcom.github.android-26utm-5Fcampaign-3Dnotification-2Demail&d=DwMCaQ&c=qqkkpu_zF8amsdnRZA_Et2-uNBtAipSVjV2iUVt238g&r=2FY5ZIPhxlzhT5oRG2zM_pT1HCmvyu01qaznhqAvB1KbjKkq3czivLM8QxnO9n4X&m=imXcx1zWWedaepzSdKCDRhSw4YlfVtyPvFpXgf8-2VM&s=DTS26_TwBFSSqLWL8Lj63yac_rzAkZkUDIAQ2MrblNQ&e=.

[kaiyan-sheng]

I will close this issue for now. Please feel free to reopen if you still see this problem with 7.14.1 or 7.15.0. Thank you!