search

clingen-data-model / clinvar-streams

1 stars 0 forks source link

Investigate connection from Confluent to AWS Lambda #18

Closed larrybabb closed 2 years ago

larrybabb commented 3 years ago

AWS Lambda's kafka source config is limited to SCRAM authentication method.

Investigate whether Confluent managed clusters can support SCRAM. If not, investigate alternate method for connecting a Confluent cluster to an AWS Lambda sink (intermediary kafka consumer container that async invokes lambda function).

Kyle F. - Feb 26, 2021 This is based on a request from Rao, but may be useful for other users wanting to ingest kafka streams through lambda or gcp cloud functions. Confluent has a connector for both AWS Lambda and GCP Cloud Functions, but this would take administration of the connection away from users, as we would need to add a config item for each user who wants the messages. We would also have to administer offset resetting for them.

tnavatar commented 3 years ago

https://docs.confluent.io/cloud/current/connectors/cc-aws-lambda-sink.html https://docs.confluent.io/cloud/current/connectors/cc-aws-lambda-sink.html ?

On Apr 7, 2021, at 4:16 PM, Larry Babb @.***> wrote:

AWS Lambda's kafka source config is limited to SCRAM authentication method.

Investigate whether Confluent managed clusters can support SCRAM. If not, investigate alternate method for connecting a Confluent cluster to an AWS Lambda sink (intermediary kafka consumer container that async invokes lambda function).

Kyle F. - Feb 26, 2021 This is based on a request from Rao, but may be useful for other users wanting to ingest kafka streams through lambda or gcp cloud functions. Confluent has a connector for both AWS Lambda and GCP Cloud Functions, but this would take administration of the connection away from users, as we would need to add a config item for each user who wants the messages. We would also have to administer offset resetting for them.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/clingen-data-model/clinvar-streams/issues/18, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC5MMC2EV6W3CYNQBNNSVDTHS4RVANCNFSM42ROXVWQ.

larrybabb commented 3 years ago

@theferrit32 no immediate action needed unless you tell me that this is a priority for Stanford (Rao). Otherwise let's let this percolate until there is a greater demand.

larrybabb commented 2 years ago

closing this. outdated. Rao is no longer at Stanford and I'm confident that Karen Dalton is not actively planning on this work. If it comes back in the future we will open a new ticket to assist Stanford with this plan.