[8.14](backport #39742) Update x-pack/metricbeat and x-pack/filebeat pipelines

[mergify[bot]]

Proposed commit message

• Updated x-pack/metricbeat pipeline to match Jenkins steps and common styling
• Updated cloud env setup script (enabled TF setup retries as originally was on Jenkins)
• Optimized cloud test steps (more details - https://github.com/elastic/ingest-dev/issues/3399)
• Enabled AWS tests despite those were disabled on Jenkins (since tests are successful)
• Applied similar cloud env setup for x-pack/filebeat && optimized cloud steps as well

Additonally:

• x-pack/metricbeat macOS arm64 tests still fail and remain disabled as it was on Jenkins, related issue created: https://github.com/elastic/ingest-dev/issues/3400
• x-pack/filebeat - updated cloud test step skip related issue

Jenkins vs Buildkite: https://gist.github.com/oakrizan/97eabb0635ad7727ae2b37e3f8469ff8

Checklist

• [x] My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ] .buildkite/x-pack/pipeline.xpack.metricbeat.yml
• [ ] .buildkite/x-pack/pipeline.xpack.filebeat.yml
• [ ] .buildkite/scripts/initCloudEnv.sh

Related issues

• Relates: https://github.com/elastic/ingest-dev/issues/3254, https://github.com/elastic/ingest-dev/issues/3399

Logs

BK builds: https://buildkite.com/elastic/beats/builds?branch=elastic%3Amergify%2Fbp%2F8.14%2Fpr-39742

---

This is an automatic backport of pull request #39742 done by Mergify.

[oakrizan]

/test