Organize Data Lake file structure to match real-world scenarios

CDCgov / data-exchange-fhir

Enterprise Data Exchange (DEX) is a new cloud-native centralized data ingestion, validation, and observation service scoped for common data types (HL7, FHIR, CDA, XML, CSV) sent to the CDC. It helps public health stakeholders who send data to the CDC while reducing the maintenance efforts, complexity, and duplication of ingestion points to CDC.

Apache License 2.0

10 stars 1 forks source link

Organize Data Lake file structure to match real-world scenarios #127

Open briansok71 opened 1 year ago

briansok71 commented 1 year ago

The team is currently just dropping all FHIR resource data as files into the root folder of the Data Lake blob container. The Data Lake container needs to be organized with folders in preparate for analytical queries. This will enable us to enhance our exporting process where data is flattened appropriately.

briansok71 commented 1 year ago

A new blob container called "fhir-data" was created, along with a folder for "bundle" and "patient". Exported FHIR resources will be uploaded to the appropriate folder in preparation for analysis.