konigio / konig

A collection of tools for semantic modeling
4 stars 4 forks source link

Filter BQ source by konig:modified in ETL pipeline #1613

Closed gmcfall closed 5 years ago

gmcfall commented 5 years ago

If a BigQuery table serves as a data source in an ETL pipeline and that table contains a column for for the konig:modified property, then I want the ETL pipeline to filter the rows so that it contains only the most recent record.

Acceptance Criteria

For now, it is sufficient to implement this capability only for the case of multiple BigQuery data sources. Later, we'll need to address the case of a single BigQuery data source.