[FEATURE] Add data-prepper ingestion pipeline for each integration

[wanghd89]

Is your feature request related to a problem? As a customer, I do not know how this dashboard can be used in real world

What solution would you like? Create end to end samples for each ingestions, so that I can try the samples by myself

What alternatives have you considered? N/A

Do you have any additional context?

1. Leverage the AWS Solution (CLO) to complete the end to end ingestion https://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/aws-services-logs.html

Or

2. Leverage the opensource Opensearch data-prepper for endto end samples https://github.com/dlvenable/data-prepper-samples/blob/master/samples/s3-source-vpc-s3-sink/data-prepper/pipelines/vpc-flow-log-pipeline.yaml

Please note: both suggestions need to tweak based on the SImple Schema.

[YANG-DB]

great idea !! @wanghd89 can u plz create an initial PR with a README file describing these use cases ?

[YANG-DB]

Specific resources

• AWS - VPC flow logs (S3 source) use the next data-prepper pipeline
• AWS ELB logs (S3 source) use the next data-prepper pipeline
• AWS Cloud Trails (S3 source) use the next data-prepper pipeline

Other examples:

• https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/grok-processor/README.md#basic-usage
• https://github.com/opensearch-project/data-prepper/tree/main/examples/log-ingestion

File based

https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/[…]pensearch/dataprepper/plugins/source/file/FileSourceConfig.java and the implementation: https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/[…]/org/opensearch/dataprepper/plugins/source/file/FileSource.java