k2v-academy / K2View-Academy

Other
0 stars 0 forks source link

TDM - populate parameters when running sync outside of TDM task #1048

Closed yBqdo2VLaCdftea1MqgSdtEhrPZtV5oJRr4eIUo closed 7 months ago

yBqdo2VLaCdftea1MqgSdtEhrPZtV5oJRr4eIUo commented 7 months ago

Hello, one of our clients has a use case to load data into TDM LU from CSV file and populate parameters so later users can do search / subsetting in TDM UI via Params option when creating Reserve task.

One of ideas was to create BW flow that will parse CSV file and execute GET for every IID in CSV, row by row (by inserting data into LU table directly from this BW flow). We can see this case we can get data populated in TDM DB for _params table (with few changes in implementation), but not in tdm_params_distinct_values (as it is populated from TDM task only ).


Best regards, Andrey

eHK5dcC00peRJzSmBkzpRSXRdBTJxvlzYiuW95S commented 7 months ago

Hi.

The task execution processes with the task execution details populate the TDM LU. The source data for the TDM LU is the TDM DB tables (tasks, task_Execution_list, task_execution_entities....). It has nothing to do with the parameters' tables. The parameters' tables (_params and the param_distinct_values) are updated and populated by the LUI sync.

Of course, it is technicaly possible to define a BF that can parse a CSV and update the TDM DB, but the correct way to populate these tables is to run an extract task on a large entity subset. If you do not want to keep these entities in Fabric, you can set the retention period to Do not Retain. You can define a custom logic that gets all entities from the CSV file and run an extract on these entities.

Tali

yBqdo2VLaCdftea1MqgSdtEhrPZtV5oJRr4eIUo commented 7 months ago

Hi Tali - in case we have CSV file as a datasource, is there a way to run 'extract' task for it? Our challenge is that we want to get all params table populated in TDM DB w/o extract task, but running GET from bw flow (from CSV parser flow)...