DataRecce / recce

The dbt data-validation toolkit for teams that care about building better data
https://datarecce.io
Apache License 2.0
252 stars 6 forks source link

[Enhancement] support merge local and cloud state #410

Closed popcornylu closed 1 month ago

popcornylu commented 1 month ago

When running recce server in the cloud mode, the syncing behavior is

  1. When launching, it load the state from the cloud
  2. When editing check, it upload(overwrite) state to the cloud
  3. When clicking check, it download(import) runs and checks from the cloud.
  4. When termingating, it upload(overwrite) state to the cloud

Problme

However, there is a case we did not cover. When run the recce server in the cloud mode. If the ci trigger the recce run --cloud and update the dbt artifacts to the code. There is no way to sync the dbt artifacts in the recce server

Solution

In this PR, we implement the merge operation. In the 2,3,4 case, it changes to merge instance of one-way upload/download. Instead, it download the state(if necessary), merge the state, and upload back to the cloud.

In addition, if user click the sync button. It would detect if there is a new state in the cloud

  1. If yes. just upload local state to remote
  2. If no, it will prompt three option to sync (merge, revert, overwrite). User can have the options to decide which way to sync with the cloud.

PR checklist

What type of PR is this? Feature/Enhacenemnt

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?: