databricks-demos / dbdemos-notebooks

Dbdemos notebook source as packaged in dbdemos toolkit
Other
12 stars 37 forks source link

Modernize The CDC/CDF (Non-DLT) Demo #13

Open alexvanadio-databricks opened 3 months ago

alexvanadio-databricks commented 3 months ago

The CDC Pipeline demo notebook is great, but it could use a little modernization. In particular:

  1. UC Support (Catalogs, Volumes, PK/FK constraints)
  2. A more realistic gold use case
  3. Better code documentation and formatting

And while we're at it:

  1. Transition to the new (V2) global setup script
  2. Remove dependencies that aren't needed (e.g. requiring an ML DBR for this demo)
  3. Minimize the amount of state scattered across %run notebooks so it's easier to reason about the code