Enterprise Data Exchange (DEX) is a new cloud-native centralized data ingestion, validation, and observation service scoped for common data types (HL7, FHIR, CDA, XML, CSV) sent to the CDC. It helps public health stakeholders who send data to the CDC while reducing the maintenance efforts, complexity, and duplication of ingestion points to CDC.
Apache License 2.0
1
stars
0
forks
source link
Parameterize consumer groups for Event Hub Loaders #176
Right now, the consumer group is derived by the topic name only. So although the consumer groups are unique by event hub, it does not allow to easily switch consumer groups.
currently, it uses;
"dbx-" + topic + "-cg-001"
We need to better parameterize this, so that we can use different event hub name patters , say we want to use cg-002 for a given topic...
So, it's best for the notebook calling the transferEventHubDataToLake to pass the consumer group name, instead of having it on the EventHubConfig
Right now, the consumer group is derived by the topic name only. So although the consumer groups are unique by event hub, it does not allow to easily switch consumer groups.
currently, it uses; "dbx-" + topic + "-cg-001"
We need to better parameterize this, so that we can use different event hub name patters , say we want to use cg-002 for a given topic...
So, it's best for the notebook calling the transferEventHubDataToLake to pass the consumer group name, instead of having it on the EventHubConfig