search

dremio / dbt-dremio

dbt (data build tool) adapter for the Dremio
Apache License 2.0
44 stars 21 forks source link

Parenthesis removal in VDS creation #167

Open Alain-Barrette opened 1 year ago

Alain-Barrette commented 1 year ago

Describe the enhancement requested

We put dbt in place for about 1,100 objects over the last month. We noticed some behavior change in the query that are executed by our users.

We traced that to the presence of encapsulating parenthesis in the object deployed thru dbt.

Going from select column from table to ( select column from table ) is causing issue when the top vds are querying. The top vds used to do

select column from tableA, tableB join on PK are now doing

select column from tableA, ( select column from tableB ) where PK join

This cause massive performance issue.

Is there any way we can remove theses parenthesis ? image

Justification for this enhancement

Query performance improvement

ArgusLi commented 1 year ago

Hi Alain, that is very strange behaviour. We will look into this as soon as we can. Thanks for raising this issue!

ArgusLi commented 1 year ago

Could you please:

  1. Let us know what version of Dremio is being used
  2. Let us know what version of the adapter is being used
  3. Provide the model (or a snippet of the model) that leads to this behaviour?

Also, could you clarify what you mean by "we noticed some behaviour change in the query that are executed by our users"? Do you mean that this was not happening before and it is now happening?

Alain-Barrette commented 1 year ago

  1. 23.1.0-202211250136090978-a79618c7

  2. fix for issue 134, base release was 1.3.2 dbt-core>=1.4.1 dbt-dremio>=1.4.1

  3. prepping model for upload

Alain-Barrette commented 1 year ago

dbt-dags-issues-167.zip

Alain-Barrette commented 1 year ago

Any update on this ?

jlarue26 commented 1 year ago

@Alain-Barrette we just finished getting dbt-dremio 1.4.5 out the door. Can you please see if you can reproduce this after upgrading?

However, dbt-dremio 1.4.5 won't automatically upgrade dbt-core. So please run pip install dbt-core==1.4.5 before pip install dbt-dremio

We believe that this regression is in dbt-core itself, but still looking into it.

Alain-Barrette commented 1 year ago

reproduced after upgrade to 1.4.5

ravjotbrar commented 1 year ago

@Alain-Barrette I made a small change in my test branch, which removes parentheses from views. Do you mind testing if this resolves your problem by cloning this repo, switching to test-branch and then installing your local repo (similar to how your team did for https://github.com/dremio/dbt-dremio/issues/134)?

Alain-Barrette commented 1 year ago

Fix from test branch is working. We are deploying to our dev team member until the fix made it to an official release. Thanks a lot everyone.