search

CDCgov / data-exchange-hl7

Enterprise Data Exchange (DEX) is a new cloud-native centralized data ingestion, validation, and observation service scoped for common data types (HL7, FHIR, CDA, XML, CSV) sent to the CDC. It helps public health stakeholders who send data to the CDC while reducing the maintenance efforts, complexity, and duplication of ingestion points to CDC.
Apache License 2.0
10 stars 14 forks source link

hl7_structure_silver is returning duplicate records on DEV #724

Closed nadiackhan closed 1 year ago

nadiackhan commented 1 year ago

hl7_structure_silver is returning duplicate records on DEV

Query: *Select FROM hive_metastore.ocio_dex_dev.hl7_structure_silver -- order by provenance.file_timestamp desc where provenance.ext_original_file_name like '%4899ba3b-4662-4d86-96a0-be297290cbef%'**

image.png

mscaldas2012 commented 1 year ago

Lets try to replicate this in TEST to validate is an actual bug and not some local function running on developers machine concurrently to the cloud function - processing the Event hub twice.

nadiackhan commented 1 year ago

Validated in Test. Records are not duplicated.

image.png