Tie confluent cloud topic state management into dev/stage clinvar-streams deployments

clingen-data-model / architecture

2 stars 0 forks source link

Tie confluent cloud topic state management into dev/stage clinvar-streams deployments #17

Open theferrit32 opened 3 years ago

theferrit32 commented 3 years ago

ccloud cli can be installed in a thin deployment startup image and have credentials passed in via secret. Either a ~/.netrc file or pass into ccloud login invocation.

These should only happen on dev/stage since topic state on prod cannot just be wiped.

clinvar-raw: delete/recreate clinvar-raw topic before launching deployment
clinvar-combiner: delete/recreate clinvar-combined topic before launching deployment

theferrit32 commented 3 years ago

It looks like the only way to authenticate to the ccloud CLI is through Confluent admin accounts, which are email-based. I don't see a way to authenticate to it using a Confluent cluster service account. We could potentially use the clingen devs google group account. Or create another account if that one is too broad use.

theferrit32 commented 3 years ago

The java org.kafka API for managing topics seems a little not-straightforward. The confluent kafka library in python has an admin client that can do it

https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.admin.NewTopic