ActianCorp / actian_tableau_connector

Tableau connector (aka taco) for Actian Avalanche, Vector, and Ingres
https://extensiongallery.tableau.com/products/936
Apache License 2.0
1 stars 4 forks source link

Review TDVT results differences between JDBC & ODBC connectors #25

Open hab6 opened 1 year ago

hab6 commented 1 year ago

Testing of JDBC and ODBC tacos built with recent changes to the connector code produces results that are very close, although not identical, between the two tacos.

Connector Passed Failed Total Tests Total Time
JDBC 819 46 865 50.13s
ODBC 823 42 865 25.78s

We need to investigate and compare the results, ensuring at a minimum that all of the tests passing for the ODBC connector are also passing for the JDBC connector.

  1. error/pass counts - priority get ensure all passing ODBC tests also pass for JDBC
  2. performance - potentially a new/separate issue - may need to pull in additional team members once we have a handle on this.
clach04 commented 1 year ago

@hab6 I've opened https://github.com/ActianCorp/actian_tableau_connector/issues/27 which maybe relevant, specifically the "Changes / TODO notes that need follow up" section.

clach04 commented 1 year ago

Those timings are a concern, 🐢 100% slower 😱

hab6 commented 1 year ago

For a quick sanity check, I ran the TDVT suite a few times with the JDBC and ODBC connectors of release 1.0.2. The ODBC connector was more than twice as fast as the JDBC connector.

JDBC ODBC
50.98s
49.47s
49.39s
24.78s
22.18s
21.39s
Avg
49.95
Avg
22.78

Investigating TDVT results with these priorities:

  1. Error/pass counts
  2. Performance
hab6 commented 1 year ago

These are the 4 TDVT tests that fail when using the Actian JDBC connector and yet do NOT fail when using the Actian ODBC connector.

TestName TestPath Test Case
date.datename.sow.week setup.date.datename.sow.week.txt DATENAME('week', [date2], 'sunday')
date.datename.sow.week setup.date.datename.sow.week.txt DATENAME('week', [datetime0], 'sunday')
date.datename.week setup.date.datename.week.txt DATENAME('week', [date2])
date.datename.week setup.date.datename.week.txt DATENAME('week', [datetime0])

Test Set: expression.standard.mahds TDSName: cast_calcs.mahds GitHub Location of Failing Tests: https://github.com/tableau/connector-plugin-sdk/tree/master/tdvt/tdvt/exprtests/standard

hab6 commented 1 year ago

Test Results with the latest TDVT suite

JDBC Connector v1.0.3(dev) Target System Passed (Rate) Failed Total Tests Avg Time
Ingres 11.2 local Windows 823 (95%) 45 868 51s
Ingres 11.2 remote Ubuntu 820 (94%) 48 868 2m 35s
Vector 6.3 remote Ubuntu 831 (96%) 37 868 1m 25s
ODBC Connector v1.0.3(dev) Target System Passed (Rate) Failed Total Tests Avg Time
Ingres 11.2 local Windows 823 (95%) 45 868 26s
Ingres 11.2 remote Ubuntu 818 (94%) 50 868 9m
Vector 6.3 remote Ubuntu 831 (96%) 37 868 4m