kromerm / adfdataflowdocs

Azure Data Factory Data Flow Documentation
196 stars 273 forks source link

complex expressions very slow #19

Open SiggyRadiation opened 5 years ago

SiggyRadiation commented 5 years ago

I have a somewhat complex expression and it causes extreme slowness in cases like these:

This is the expression:


iif (instr(code_bekostigingstatus, 'pi') > 0 ,'', 
iif (instr(code_bekostigingstatus, 'jl') > 0, 'Geen 1e inschrijving', 
iif (instr(code_bekostigingstatus, 'nr') > 0, 'Nationaliteiteneis',
iif (instr(code_bekostigingstatus, 'nb') > 0, 'Eerdere graad behaald',
iif (instr(code_bekostigingstatus, 'nd') > 0,'Eerdere graad behaald',
iif (instr(code_bekostigingstatus, 'nh') > 0, 'Bekostigingsjaren verbruikt',
iif (instr(code_bekostigingstatus, 'ni') > 0, 'Bekostigingsjaren verbruikt',
iif (instr(code_bekostigingstatus, 'nk') > 0, 'Bekostigingsjaren verbruikt',
iif (instr(code_bekostigingstatus, 'ti') > 0, 'Geen tijdige aanlevering',
'Anders')))))))))

Fyi the logic behind this is that the field code_bekostigingsstatus can contain multiple codes seperated by a comma. The organisation only wants to know the meaning of most highest priority code that is contained in the field, so I scan the field from highest to lowest priority code.

image

kromerm commented 5 years ago

How many columns are in your source data?

I'd like to have our Engineering team look at this. Can you send us an email directly? Email is adfdataflowext@microsoft.com.

Go to the Data Flow designer UI and click the Code button at the top right, then choose "Plan".

Copy & paste that script into the email with your problem description from above.

Thank you!! Best, Mark Kromer