BUG - fivetran dbt-netsuite package 0.3.3 perfomance issue

[rdnxls]

Are you a current Fivetran customer? Yes

Describe the bug Hello,

After upgrading fivetran dbt-netsuite package from 0.2.3 to 0.3.3 we are constantly experiencing perfomance issues. The runtime increased from 2 minutes to 2-3 hours in average.

Best, Alexander

Steps to reproduce dbt run

Expected behavior Runtime less then 5-10min

Project variables configuration

copy dbt_project.yml here

Package Version models: netsuite: +schema: # leave blank for just the target_schema netsuite_source: +schema: # leave blank for just the target_schema vars: transactions_pass_through_columns: ['transaction_extid']

packages:

• package: fishtown-analytics/dbt_utils version: 0.6.4
• package: fishtown-analytics/codegen version: 0.3.1
• package: fishtown-analytics/logging version: 0.4.2
• package: fivetran/netsuite version: 0.3.3

**Warehouse**
<!--- Which warehouse you are running dbt and this package against. -->
<!--- To check a warehouse remove the space and insert an x in the box (eg. [x] Bigquery). --> 
- [ ] BigQuery
- [ ] Redshift
- [ x] Snowflake
- [ ] Postgres
- [ ] Databricks
- [ ] Other (provide details below)

**Additional context**
<!--- Add any other context about the problem here.  --> 

**Screenshots**
![Screenshot 2021-08-25 at 15 45 08](https://user-images.githubusercontent.com/73248144/130821154-1f897255-c6d7-473e-b6f2-664d990e30ce.png)

**Please indicate the level of urgency** 
high - blocked from upgrading dbt 0.2.0

**Are you interested in contributing to this package?** 
<!--- Our packages are all open-source and we welcome contributions from the community! Are you interested in helping us fix this bug? -->
- [ ] Yes, I can do this and open a PR for your review.
- [ ] Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed. 
- [x ] No, I'd prefer if someone else fixed this.  I don't have the time and/or don't know what the root cause of the problem is.

[fivetran-joemarkiewicz]

Hi @asargumed thank you for opening this issue and that is extremely strange that the runtime jumped from 2 minutes to over 2 hours?! I will look into this and see what updates could possibly be causing this performance blocker.

[fivetran-joemarkiewicz]

@asargumed I have looked through the code differences between v0.2.3 and v0.3.3 and do not see any erroneous joins or queries that would potentially cause performance to struggle as much as you are showing. Would you be open to setting up some time during out dbt package office hours so we may look into this together live?

[rdnxls]

@fivetran-joemarkiewicz Today the issue disappeared and the runtime is stable on 1 min. Therefore I would cancel the appointment today. Since I can't reproduce the issue and will contact you when it appears again. Sorry for the short notice and thank you for your support

[rdnxls]

Hi, the issue is back

[fivetran-joemarkiewicz]

Hi @asargumed I am sorry to hear the issue is still persisting. Would you be able to share what the size of your Snowflake warehouse is? Outside of this package I have experienced extremely long query performance issues on my Small warehouse size.

Do you notice that these queries tend to take longer at specific periods of the day?

[rdnxls]

Hi Joe, unfortunately I overseen your message. We are using a small warehouse with 1 cluster, since we migrated to Netsuite this year do not have much data there.

After materializing all objects as tables the issue disappered. Im not sure what caused the issue, but its gone now again.

Unfortunately I couldnt recognise any specific trend, when the perfomance issue occured. But usually, when it appeared, we were basically blocked for the whole day.

I will close the issue and cancel the appointment and let you know it comes back.

Best regards and thank you for support, Alexander