Originally raised in https://github.com/meltano/squared/pull/351#issue-1336530069. Snapshots arent being fully tested if they are brand new in their isolated CI environment i.e. theres no inactivated records and its basically a "select *" vs a merge. Without this theres no historical inactivated records in snapshots during CI.
In CI we can clone the production snapshots prior to running the EL tests. That way we pull new data and snapshot that against the existing snapshot.
add a dbt macro that clones snapshots from prod to isolated CI env
convert CI dbt_seedjob to more generic dbt_prep, call clone macro from this
Other notes:
this unlocks implementing activated dates for IP addresses. Right now we only filter for inactivated dates. Either implement or create and issue for that.
theres a challenge here when a snapshot is new and not in production, maybe the clone script does a check and only clones if it exists. Otherwise it will fail on the first CI run.
pnadolny13
commented
1 year ago
Originally raised in https://github.com/meltano/squared/pull/351#issue-1336530069. Snapshots arent being fully tested if they are brand new in their isolated CI environment i.e. theres no inactivated records and its basically a "select *" vs a merge. Without this theres no historical inactivated records in snapshots during CI.
In CI we can clone the production snapshots prior to running the EL tests. That way we pull new data and snapshot that against the existing snapshot.
dbt_seedjob to more genericdbt_prep, call clone macro from thisOther notes: