search

clingen-data-model / data-exchange-topics

For issue tracking and managing configurations of kafka topics in the dx
0 stars 0 forks source link

LDH-VP topic conversion to single partition (dev/prod) #33

Open larrybabb opened 2 years ago

larrybabb commented 2 years ago

Topic Producer/Developer Contact Details

Mark Mandell & Kevin Riehle

Producer API Key

Baylor's API key for Producer, Stanford's API key for Consumer

Topic Consumer/User Information

LDH, VP

Topic Name (suggested)

TBD (see https://clingen.slack.com/archives/C018FL85SN8/p1654274355759549)

Topic Description

@Kevin Riehle … @Kyle Ferriter, @Karen Dalton, @Mark Mandell and I just met and determined we need to reset the LDH to VP topic on the dev cluster to be a single partition (vs the 20 partitions it is configured as now). We also will set up a single partition topic with the same name on the prod cluster. Here are the steps that we all need to work on as a group to get to the end state of both a dev and prod topic loaded with LDH data to feed the VP consumer process.

  1. @Kevin Riehle and @Mark Mandell provide @Kyle Ferriter the final topic name for the dev and prod cluster. If you want to reuse the existing topic name then we will have to rename that since we plan to keep that intact until we have the new topic setup.
  2. @Kyle Ferriter will setup the single partition topics on prod an test and verify that both Baylor and Stanford have the appropriate read/write permissions to being using it. He will then inform you on this channel when it is ready.
  3. @Kevin Riehle and team will reload the new single partition topic and the new single partition prod topic with the appropriate data from the LDH source systems. @Kevin Riehle will notify @Mark Mandell on this channel when this is complete.
  4. @Mark Mandell will consume the data and determine that the old multi-partition topic is no longer needed. Once this determination happens, @Mark Mandell will notify @Kyle Ferriter on this channel and he will delete the multi-partition topic on dev

@Kevin Riehle I will leave it to you to work with @Karen Dalton and @Mark Mandell to get step 3 above accomplished at the earliest possible convenience. (If you need to break it up and do dev only first and then defer prod then that is a discussion for Baylor and Stanford).

Number of partitions

1 (recommended)

Message Persistence

Indefinitely (yes)

Can the data in this topic be shared publicly?

Not yet.

Which clusters should this topic be created on?

Do you want to have the topic backed up to GCS?

larrybabb commented 2 years ago

Kevin do you want this stream backed up or are you willing to take responsibility for recreating it in the event that is ever needed?

theferrit32 commented 2 years ago

Awaiting update from Baylor and Stanford teams about when they will be ready to switch to the prod topic, and whether they will replace the content on the dev/stage topics.

theferrit32 commented 2 years ago

Moving out of in-progress because there is additional information needed.

larrybabb commented 2 years ago

@kriehle just giving you a heads up in case there's any updates on moving off of the 20-partition topic to the new single partition topic.