datafold / data-diff

Compare tables within or across databases
https://docs.datafold.com
MIT License
2.95k stars 270 forks source link

Support different where filter for source table and target table #889

Closed Schumpeterx closed 5 months ago

Schumpeterx commented 6 months ago

describe. For different database the where filer may different. Blow is an exmple:

In postgresql we use date_trunc('month', cast(current_date as timestamptz)- interval '0 month') to get the first day of month . While in mysql the sql should be cast(date_add(curdate(),interval -day(curdate())+1 day)as datetime).

Describe the solution you'd like Adding source_where and target_where to the configuration file may be a good choice

[run.test]
  # Source 1 ("left")
  1.database = "mysql"
  1.table = "test_time"

  # Source 2 ("right")
  2.database = "pdb"
  2.table = "test_time"
.................
  key_columns = ["id"]
  columns = ["time_col"]
  source_where="time_col!='2024-01-01 00:00:00'"
  target_where="time_col!='2024-01-02 00:00:00'"
glebmezh commented 5 months ago

Hi @Schumpeterx,

Thank you for trying out data-diff and for taking the time to open this issue. We made a hard decision to sunset the data-diff package and won't provide further development or support. Diffing functionality will continue to be available in Datafold Cloud. We have completely rewritten the diffing engine in the cloud over the past few months and have solved the fundamental issues with the original algorithm used in the data-diff package. Feel free to take it for a trial or contact us at support@datafold.com if you have any questions.

-Gleb