The SDGym benchmark script currently allows users to run a quality report for each dataset/synthesizer pair. But it does not allow them to run the diagnostic report.
The diagnostic report (as implemented in SDMetrics) checks for overall data validity between the real and synthetic data. For the health of a synthesizer, it is important to verify that the synthetic data it produces always has a score of 1.0.
Expected behavior
Add a parameter called compute_diagnostic_score to the benchmarking script.
(default) True: Run a diagnostic report for each synthesizer/dataset pair being evaluated and report the score. The results should appear in as a new column called Diagnostic Score. This column should appear before the Quality Score column.
False: Do not run a diagnostic report
Additional context
When run, the time it takes to compute the diagnostic report should be included in the overall Evaluate Time.
Problem Description
The SDGym benchmark script currently allows users to run a quality report for each dataset/synthesizer pair. But it does not allow them to run the diagnostic report.
The diagnostic report (as implemented in SDMetrics) checks for overall data validity between the real and synthetic data. For the health of a synthesizer, it is important to verify that the synthetic data it produces always has a score of 1.0.
Expected behavior
Add a parameter called
compute_diagnostic_scoreto the benchmarking script.True: Run a diagnostic report for each synthesizer/dataset pair being evaluated and report the score. The results should appear in as a new column calledDiagnostic Score. This column should appear before theQuality Scorecolumn.False: Do not run a diagnostic reportAdditional context
When run, the time it takes to compute the diagnostic report should be included in the overall
Evaluate Time.