AWS focused documentation improvement

ravikesarwani commented 2 years ago

Problem

AWS is the largest cloud provider by any means. We as a company are focused on Cloud and many of our largest customers are also running their workloads in AWS. While we have many ways of data ingestion, there’s no single landing documentation page to drive the data ingestion experience for the AWS observability (& security - logs for SEIM) customers.

Currently we have many ways to ingest data

Beats:
- Filebeat & Metricbeat and various modules
- Functionbeat - older method for serverless data ingestion (logs)
Elastic agent and various integrations
Elastic-serverless-forwarder - a new method for serverless data ingestion (logs)
Firelens (no code but provided a sample task definition that can be used by customers as a starting point to set things up)
AWS Lambda APM monitoring

All of this documentation lives in different places and isn't connected well.

Existing resources

These are all of the documentation pages we currently have that are related to AWS monitoring:

Amazon Web Services (AWS) monitoring. A stub page that has subpages for monitoring these AWS resources:
AWS integration docs (docs.elastic.co)
Two tutorials:
- Monitor AWS with Elastic Agent
- Collect VPC flow logs and S3 access logs from AWS.
- Collect billing and EC2 metrics from CloudWatch.
- Install and configure Elastic Agent to stream the logs and metrics to Elasticsearch.
- Visualize your data in Kibana.
- Monitor AWS with Beats
- Create and configure an S3 bucket
- Create and configure an SQS queue.
- Install and configure Filebeat and Metricbeat to collect Logs and Infrastructure metrics
- Collect logs from S3
- Collect metrics from Amazon CloudWatch
Elastic Serverless Forwarder for AWS
Monitoring AWS Lambda functions
Beats
- Filebeat AWS module
- Metricbeat AWS module
- Functionbeat (replaced by Elastic Serverless Forwarder)

Other resources

These resources could be useful when working on this ticket. We'll likely want to move some of this blog/pdf content to the docs.

What we need

Main goals:

A single landing page in the Observability Guide for AWS Observability customers that guides them through the data ingestion process.
A focus on explaining the various ingestion methods, their pros and cons, and why a customer would use one over the other
A focus on getting started/onboarding with data ingestion

Other goals:

We need to provide more detail about why users should watch specific metrics (similar to what we do in the EC2 topic)
Where we list the metrics to watch, we should tell users how to view them in the UI
This guide should also link to AWS security solution materials

This ticket is a complete restructuring and rethinking of how and where we currently document AWS monitoring.

Thoughts and questions

Long ago, a decision was made to focus on Elastic Agent in the docs (over Beats). In recent months, we've moved away from this and started to explain both the Beats/Agent methods, including pros/cons. An example can be seen in our log ingestion guide. It sounds like this is the approach we should take, but we should verify that before moving forward.
Right now we have two different tutorials for AWS monitoring. Does this make sense if we're going to continue to support both Beats and Agent?
Should we move the non-reference content out of the AWS integration docs? Moving this content to the Observability Guide could make it easier to maintain.
How do the changes we make here apply to our documentation for monitoring other Cloud providers? For example, can we apply the same restructuring/rewriting to GCP? Azure?
This related issue may apply after a plan is devised: https://github.com/elastic/ingest-docs/issues/103

ollyhowell commented 2 years ago

Thanks @ravikesarwani

ravikesarwani commented 2 years ago

@ollyhowell Have you given some thoughts on the next steps and how/when we can start to collaborate on this?

ollyhowell commented 2 years ago

Hi @ravikesarwani sorry, nothing concrete yet but we're thinking about placement within guide and will get in touch with you next week - still working thru some 8.0 and 8.1 issues that are not closed yet so bandwidth is limited. Hopefully we can get the ball rolling for 8.2 with a small update though...let's discuss.

ravikesarwani commented 2 years ago

cc: @pmeresanu85 FYI

pmeresanu85 commented 2 years ago

@ollyhowell any way we can pick this up again? Anything your team needs from us?

@ravikesarwani maybe an opportunity to works towards a "How to monitor AWS with Elastic Observability guide?"

ravikesarwani commented 2 years ago

We have an old AWS observability guide https://www.elastic.co/pdf/the-elastic-observability-guide-for-aws.pdf that's very outdated. @ollyhowell We are looking for something like that but with few differences:

Focus more on metrics for all the AWS service Elastic Agent integrations we have
Focus on "value" to the customers detailing the services specific metrics that we collect and how to make sense of that data in the dashboards.

ollyhowell commented 2 years ago

Thanks for the comments and offers - some hopefully useful points from my side:

we're working on a Data Ingest guide that will cover ingestion across all solutions and pipelines; this will provide the best home for all AWS ingestion topics e.g. plan is to have a How To | Start ingesting data | Ingesting AWS data topic. Workgroup has kicked off and we're planning tasks for 8.5 and beyond but we'll loop you in next week
I do have a draft issue for 'mining' that AWS pdf already but it's stuck in the backlog (hold on, let me convert it: https://github.com/elastic/observability-docs/issues/2157) - it might be a good task to prioritise in that there is a lot of useful content in it (I hope!) and converting and updating should be reasonably formulaic and something that could be started within dev team, and it sounds like you have a clear idea of what needs changed
the AWS tutorial within Obs guide is also being updated to focus on Agent over Beats (https://github.com/elastic/observability-docs/issues/2091) this will need a PM review but we could potentially incorporate other improvements too if you have strong opinions here

ollyhowell commented 1 year ago

Linking the draft PR for serverless forwarder: https://github.com/elastic/observability-docs/pull/2214

dedemorton commented 1 year ago

Noting here that there's a PR open with some initial docs about AWS monitoring. It does not address all the issues related to our fragmented ingest story, but will hopefully help get users started with AWS integrations. https://github.com/elastic/observability-docs/pull/2564

dedemorton commented 10 months ago

Moving this to the next sprint because the focus of the current sprint has been on serverless.

For the next sprint, I'll follow up to scope the work and figure out what is required here, but I won't have time to work on the actual documentation before the new year.

alaudazzi commented 7 months ago

@ravikesarwani @pmeresanu85 I'm taking this up for the next sprint starting Feb 24. I'll go through the comments in this ticket and get some historical background to figure out what is required. It would be useful to connect one of these days to better frame the scope of the work.

vinaychandrasekhar commented 7 months ago

@alaudazzi Please work with @katrin-freihofner and @SubhrataK on this topic.

Ravi and Paul no longer work at Elastic. Thanks

bmorelli25 commented 7 months ago

@alaudazzi I've updated the issue description to (hopefully) better define the goals of this issue.

alaudazzi commented 7 months ago

@bmorelli25 A few more thoughts and questions:

Long ago, a decision was made to focus on Elastic Agent in the docs (over Beats). In recent months, we've moved away from this and started to explain both the Beats/Agent methods, including pros/cons. An example can be seen in our log ingestion guide. It sounds like this is the approach we should take, but we should verify that before moving forward.

Who should we verify with? Who are the main stakeholders in this project?

Right now we have two different tutorials for AWS monitoring. Does this make sense if we're going to continue to support both Beats and Agent?

If the two tutorials you refer to are the ones mentioned in the root comment (Monitor AWS with Elastic Agent and Monitor AWS with Beats) then I would say it makes sense to maintain them.

Should we move the non-reference content out of the AWS integration docs? Moving this content to the Observability Guide could make it easier to maintain.

Assuming that all the parties involved (Integrations + Security teams) agree with the idea of keeping the Integrations docs more as reference material.

How do the changes we make here apply to our documentation for monitoring other Cloud providers? For example, can we apply the same restructuring/rewriting to GCP? Azure?

It would be nice to keep some sort of consistency.

alaudazzi commented 7 months ago

I met with @katrin-freihofner -- here are the main points from our meeting:

Brainstormed on what we have and what we should have, based on the input provided in this ticket
Looked at the initial skeletons drafted by Arianna and discussed how these might evolve
Emphasized the need for promoting Firehose and ESF as ingestion options
The upcoming EAH might be a great opportunity to get together with Baptiste Turquet and Andrew Wilkins and further discuss the WIP doc structure

Arianna reached out to the Obs support engineer Luca Belluccini to engage them in the doc reorg and get input where required

alaudazzi commented 7 months ago

I met with @lucabelluccini -- main points from our meeting:

integrate into the doc the table where we compare the various ingest methods
be more specific about the types of logs and metrics collected with various ingest options
add the Scaling model, similar to what we already have in Azure and GCP
touch base with:
- Kaiyan-sheng (engineering) to validate the structure and content
- Louis Ong (support) who worked on many AWS KB articles

Update the AWS doc redesign accordingly.