Extraction process is getting staled everytime

[fran-puebla]

Hi Darren, launching the extraction process we are not able to finish one execution since the process is getting staled every time in the same object (tdxldtec_cal_iabono_8), even excluding the object we got the same result.

We already enabled debug functionality but no clues there neither.

We are not sure if the problem related to this specific object or maybe something related to memory assigned, we are little bit lost here.

Total views: 2343 Object where process is getting staled: 1833

Log execution not filtering:

            sabadell_des.tdxldtec_cal_freqcobrocomis TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_freqliq TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_garantiacredi TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_grupoppalriesgo TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_gruporiesgo TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_gruporiesgore TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_horpref TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_iabono_1 TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_iabono_2 TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_iabono_3 TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_iabono_4 TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_iabono_5 TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_iabono_6 TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_iabono_7 TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_iabono_8 TABLE_TYPE=2 columns: 5 expressions:0

Log execution filtering out exclude.datasets=.tdxldtec_caliabono

            sabadell_des.tdxldtec_cal_freqliq TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_garantiacredi TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_grupoppalriesgo TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_gruporiesgo TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_gruporiesgore TABLE_TYPE=2 columns: 5 expressions:0
            sabadell_des.tdxldtec_cal_horpref TABLE_TYPE=2 columns: 5 expressions:0
                    excluding object from scan sabadell_des.tdxldtec_cal_iabono_1
                    excluding object from scan sabadell_des.tdxldtec_cal_iabono_2
                    excluding object from scan sabadell_des.tdxldtec_cal_iabono_3
                    excluding object from scan sabadell_des.tdxldtec_cal_iabono_4
                    excluding object from scan sabadell_des.tdxldtec_cal_iabono_5
                    excluding object from scan sabadell_des.tdxldtec_cal_iabono_6
                    excluding object from scan sabadell_des.tdxldtec_cal_iabono_7
                    excluding object from scan sabadell_des.tdxldtec_cal_iabono_8

As you can see, no error, no message.

Any insight you can give us?

[darren-wrigley]

Hi @fran-puebla i apologize for taking so long to get to this. i think i have just found where the issue is & am working on refactoring the code to workaround some limitiations that appear to be in Denodo. what was happening is when there are >1000 views in a single denodo database (schema in EDC), then when trying to get the next view from a get_views query, the system just hung (denodo never returned anything and did not throw and error).

the workaround will be to process in chunks of 1000, using limit and offset parameters.

i just finished an initial test and I was able to re-produce the problem. once i finish a complete test on all known examples, i will update the code here and the package on Informatica Marketplace

[darren-wrigley]

issue should be resolved now, using v1.1.000