Schema Gathering and Data Gathering Phases

itd-sk commented 1 year ago

We are trying to migrate our WDC Connector v2 to the new v3 SDK. We have multiple data tables and some tables require an additional retrieval request to get the schema. We are now wondering how we can split the gathering phase into a schema gathering and data gathering phase. The FetchOptions contain the properties gatheringPhase and isSplittingGatheringPhase but the gatheringPhase is always "SchemaGathering" and isSplittingGatheringPhase is always false. Is there an option to set that the gathering phases are split? Or is there documentation when the Fetcher.fetch is called with which options and how are the best practices to fetch which data when (schema data and table rows)?

itd-sk commented 8 months ago

Is this now possible with version 2 of the taco toolkit (https://www.npmjs.com/package/taco-toolkit) ? I can't find a detailed documentation what has been changed. Without the two data gathering phases "schema" and "data" we would need to load all data of a customer's system even if only a small part of the data is used by a Tableau report. This would mean that there is still no way to migrate from WDC 2 to WDC 3.

ataft commented 8 months ago

For reference, I'm posting the new NPM package link with versions (2.0.0 released Oct 12, 2023): https://www.npmjs.com/package/@tableau/taco-toolkit?activeTab=versions

Indeed, the community would love some documentation.

daubli commented 7 months ago

I have the same problem. It seems like Tableau calls the *fetch-method for every registered handler. This happens when you call connector.submit() but also when you try to build relationships between tables. For every join you make Tableau refreshes the whole dataset. Then if you want to take a look into an arbitrary table...Tableau fetches again all tables. This is annoying and I spent three days to find a way around that without success. The WDC Connector v2 did a significant better job there.

itd-sk commented 7 months ago

@bcantoni Could you please give us the information if this will be implemented in the future? Our customer's are wondering if we can provide a Tableau connector in the future. In general, the current situation is very unsatisfactory:

Web Data Connector has been marked as deprecated in the product with no real alternative.
It is unclear when it will no longer be possible to use a Web Data Connector.
It is not possible to publish a Tableau report with a WDC 3 connector as data source on Tableau Online if the connector is not certified. However, there is no information on how the connector can be certified.
There is no contact person for this migration topic. The communication here and in the community forum is very poor to non-existent.

repl-kannan-d commented 5 months ago

I have the same problem. It seems like Tableau calls the *fetch-method for every registered handler. This happens when you call connector.submit() but also when you try to build relationships between tables. For every join you make Tableau refreshes the whole dataset. Then if you want to take a look into an arbitrary table...Tableau fetches again all tables. This is annoying and I spent three days to find a way around that without success. The WDC Connector v2 did a significant better job there.

Yes, i am also seeing the same issue. This is very annoying and also concerning.
1)Dataset in our customer db will be very large(million rows in single table) and in WDC2 we are loading 37 tables and out of it atleast 10 tables will contain huge data. This means that we never be able to migrate to WDC3 as tableu will get crashed even on loading the connection page(schema). 2)Even to use single table data, tableu loads all tables data - This makes unnecessary load on our server with no use and doesn't make sense.

It will be good to know if there is a way to control thing behaviour

tableau / wdc3

Schema Gathering and Data Gathering Phases #3