System interface: DEX-EDAV

[dcdlz]

Our intent is that CDC programs will consume DEX ingested and processed data via EDAV.

Summary of child issues

• [X] EDAV Subscription ADLS accounts #71, #117
  • ~General Storage Account~ Revoked by EDAV 2022-10-07
  • DEX Storage Account
• [x] DEX Subscription ADLS to Databricks: #24
• [x] Grafana to Databricks: #92
• [x] DEX EventHub to Databricks: #159 - [ ] Databricks to SQL Endpoint: We are not supporting SQL consumption pattern

ADLS

Auth	EDAV General Storage Account (Dev)	EDAV DEX Storage Account (Dev)
Azure Portal	X	X
AD credentials from CLI	X	X
SAS tokens	X	X
Service Principal

Open questions

• [ ] What is the EDAV Configuration of DEX Storage Account
  • Container/Folder structure
  • General, PHI, CIPSEA
  • Life-cycle policies (hot, cool, cold, archive tier)
• [ ] How does EDAV manage access to different programs?
• [ ] How do programs that aren't using Databricks get access to their data?
  • [ ] Azure Synapse --> SQL Server Management Studio (SSMS)? From @raman-cdc
  • [ ] DB Visualizer connection to EDAV Databricks? From @raman-cdc

Closed questions

• [X] "EDAV-2" flow: DEX EventHub --> EDAV EventHub ==> No. We will use DEX EventHub.
• [X] Write to program-specific spaces instead of a generic DX space within EDAV. ==> No. We will write to a DEX Storage Account within EDAV.

[rmharrison]

Appendix: Architecture decision for DEX-EDAV interface

We considered three options for the interface Source: LucidChart > Transport Component Interface > EDAV

Option 3: Selected

Option 2: Desired by team

The team wanted to do Option 2; but, our request for a DEX Databricks was denied by OCIO.

Briefly, Option 2 allows DEX to move faster because DEX will be able to prototype features before moving to EDAV Databricks

• Reduces coordination with EDAV
• We don't know all the Databricks connections we will need
• Known needed connections
  • Hookup EventHubs to Databricks (2022-09-01: EDAV explicitly said they'd allowed)
  • Azure Managed Grafana to Azure Managed Databricks Plugin (2022-09-16: Still will Microsoft support) cc @raman-cdc
  • SQL Endpoint (2022-09-01: EDAV hasn't allowed yet)

Option 1: No

Requires ADLS replication from DEX to EDAV.

• Point DX ADLS endpoint to EDAV ADLS <folder?> ==> Folder from EDAV
• Create DX subnet ranges for <writer? : ADLS, Function> within the DX VNET ==> Decide on EDX writer
• Write DX ADLS to EDAV ADLS