search

tableau / connector-plugin-sdk

SDK for Developing Tableau Connector Plugins
https://tableau.github.io/connector-plugin-sdk/
MIT License
108 stars 108 forks source link

TDVT expected results have incorrect values #1174

Closed hab6 closed 1 year ago

hab6 commented 1 year ago

About You: Name: Michael Habiger Company: Actian Corporation

Your question: I found a few cases where the TDVT expected results data appears to be incorrect.

Case 1

Test: DATENAME('week', [date2], 'sunday')' Value in question from Calcs TDVT table data date2 column: 1995-06-04 Manual check and online date calculator give a value of: 23

TDVT Expected Data File Value Evaluation
expected.setup.date.datename.sow.week.txt 23 Correct
expected.setup.date.datename.sow.week.1.txt 23 Correct
expected.setup.date.datename.sow.week.2.txt 24 Incorrect

Case 2

Test: DATENAME('week', [date2], 'sunday') Value in question from Calcs TDVT table data date2 column: 1995-09-03 Manual check and online date calculator give a value of: 36

TDVT Expected Data File Value Evaluation
expected.setup.date.datename.sow.week.txt 36 Correct
expected.setup.date.datename.sow.week.1.txt 36 Correct
expected.setup.date.datename.sow.week.2.txt n/a Incorrect

Case 3

Test: DATENAME('week', [datetime0], 'monday') Value in question from Calcs TDVT table data datetime0 column: 2004-07-04 22:49:28 Manual check and online date calculator give a value of: 27

TDVT Expected Data File Value Evaluation
expected.setup.date.datename.sow.week.txt 27 Correct
expected.setup.date.datename.sow.week.1.txt n/a Incorrect
expected.setup.date.datename.sow.week.2.txt 27 Correct
pvanderknyff commented 1 year ago

Hi Michael,

TDVT has multiple expected files to support different capabilities and formats of all the databases we test against. Especially with dates, there may be more than one "right" answer, or subtle differences in how databases calculate answers. To pass the test, you only need to match one expected file. This, the behavior you're seeing with expected files is by design, likely written years ago to cover an edge case with a certain database.

The wrinkle is that your connector needs to match all of the expected values in one of the files to pass the test. If your connector isn't passing any of them, but is giving the correct values, this may be another case for a new expected file. Attach your actual values for the test into the issue and I'll take a look. Otherwise, I think this issue can be closed.

hab6 commented 1 year ago

Thanks @pvanderknyff for your quick response and clear explanation, which really helps in understanding the TDVT process for comparing actual versus expected results.

The failures that prompted this issue look to be nuances in our dialect for which I am investigating. I can open a separate issue if it turns out that we are generating correct values for a particular test and not matching any of the sets of expected values. Therefore, I think this issue can be closed.