search

dbt-labs / dbt-codegen

Macros that generate dbt code
https://hub.getdbt.com/dbt-labs/codegen/latest/
Apache License 2.0
459 stars 99 forks source link

Improve DX with `just` #165

Closed gwenwindflower closed 4 months ago

gwenwindflower commented 5 months ago

resolves #144

This is a:

All pull requests from community contributors should target the main branch (default).

Description & motivation

codegen is awesome, but the developer experience is not the best. Due to its aim of using dbt itself to template the code, and not comprising on any sketchy hacks to write files, you're left getting command line output that you need to copy and paste into files. Many people over the years (myself included) have wrapped codegen with shell scripts, Python scripts etc to solve for this, but largely these efforts have remained isolated and haven't made their way back into this repo.

Thanks to some excellent work and thinking from @wjhrdy we're aiming to change that, starting with adding a robust set of just commands to the repo.

Using just we can wrap and chain the macros with all the power of shell scripts (and even other scripting languages!)

Checklist

gwenwindflower commented 5 months ago

@wjhrdy here's our branch to collaborate on. I've left the core recipes as they are, but have expanded with heavy commenting/documentation in the file and allowing for people to choose their package manager and related config for that. We don't offer any official documentation from dbt Labs on using poetry with dbt Core, so I don't feel comfortable with relying on that as the only path for these recipes to work, thus the added logic. Totally good with keeping that as an option though and I made sure to highlight why it's a good choice for this purpose in the commenting/documentation. I haven't fully tested that this system works, just a draft for now, but hopefully this should let people use pip by default, while still optionally going with poetry or uv (my personal preference).

Let me know if this works for you, and feel free to push commits from here! Thanks again for the amazing work on this already, looking forward to seeing where we can go with this!

Like all things, I'm very open to discussing and changing my mind about how to approach this 😸. We're trying to push people towards having per-project installs of dbt in a virtual environment, so this is in-keeping with that.

gwenwindflower commented 5 months ago

Also @dbeatty10 if you want to get in on this, since you've also got the just bug!

jasnonaz commented 4 months ago

Hi Willy,

I’m Jason - manager of the DX team here at dbt Labs. I want to thank you for all of the time and effort you’ve put into thinking through this PR and how it could improve dbt-codegen.

I was running through the upcoming release with Winnie and seeing this PR made me realize two things:

  1. This is an extremely interesting angle to investigate
  2. This is a relatively large change to the usage of dbt-codegen - one that most likely falls outside the size of change we’re looking to make in the upcoming minor release, which is mainly aimed at addressing potential fixes and improvements that fit squarely within existing functionality

For that reason, we’re going to close this as won’t do for now. It’s possible that in the future we might be looking to more holistically revamp the codegen package, at which time we’d be happy to pick this up with you.

I want to apologize for having this happen at this stage in the process - typically the best time for us to mark something as won’t do as when the initial issue is created, not after you put in a bunch of work. That’s on me for not setting the bounds correctly on this codegen release - I hope you know that we really appreciate all the care and thought you’ve put into this.

If I might make a suggestion, you’ve done a bunch of cool investigation here and this might make for an interesting #show-and-tell post on the dbt Community Forum - I think the folks there would really appreciate seeing the thinking you've done.

Thank you again,

Jason