MattTriano / analytics_data_where_house

An analytics engineering sandbox focusing on real estates prices in Cook County, IL
https://docs.analytics-data-where-house.dev/
GNU Affero General Public License v3.0
9 stars 0 forks source link

Add tasks to register new tables as Great Expectations Data Sources #141

Closed MattTriano closed 1 year ago

MattTriano commented 1 year ago

The CLI GX interface configured data connectors that essentially adds a database schema (so tables added to that schema automatically become available to GX). The GX team is deprecating the CLI interface in favor of a more pythonic workflow. I can still maintain my prior connector schema as a "block-config", but switching from a CLI interface to a more pythonic interface might enable me to automate away a rather high friction mode-change in the development process. So I'm willing to explore the new interface.

Using the new interface means each table I want to set expectations for must be explicitly registered. The update pipeline can handle automatically registering expectations. The data_raw.temp_ table and data_raw persistent tables are easy candidates. I'm not sure if it makes sense to register views, so I'll hold off on registering the _standardized and _clean views.