[BUG] bootstrap_servers is not configurable for Kafka Source in AWS

[a10pepo]

Describe the bug Creating an opensearch pipeline in AWS is not possible using the following yaml:

`version: '2' kafka-pipeline: source: kafka: bootstrap_servers:

• '100.100.100.100:9092' topics:
• name: Topic1 group_id: groupID1 sink:
  • opensearch: hosts: [ "https://XXXX.eu-central-1.amazonaws.com" ] aws: sts_role_arn: "arn:aws:iam::XXXXXXX:role/AWSROLE" region: "eu-central-1" index: "index_123"`

It returns an error with the following message:

"kafka.bootstrap_servers" is not configurable parameter for Amazon OpenSearch Ingestion pipelines using an aws kafka source: "$['log-pipeline']['source']['kafka']".

To Reproduce Steps to reproduce the behavior:

1. Go to Opensearch Ingestion Pipelines
2. Create a new pipeline
3. Paste the yaml and validate
4. See error

Expected behavior It should be valid and start a pipeline

Screenshots

Environment (please complete the following information):

• AWS eu-central-1
• OpenSearch Cluster Version 2.11 (AWS Latest)
• Service Software Version: OpenSearch_2_11_R20231113-P2 (latest)

[asifsmohammed]

@a10pepo This is supported in Data Prepper but not in Amazon OpenSearch Ingestion pipelines. For questions related to Amazon OpenSearch Ingestion please open an AWS support case.