search

codeforkjeff / dbt-sqlite

A SQLite adapter plugin for dbt (data build tool)
Apache License 2.0
78 stars 13 forks source link

Need a new maintainer for this project #48

Open codeforkjeff opened 1 year ago

codeforkjeff commented 1 year ago

Due to some personal issues, I unfortunately don't have time to maintain this project anymore. This adapter works with dbt 1.4.0 but there have been two minor versions since then that probably require changes for full compatibility.

If anyone wants to take this over, please let me know and we can coordinate it somehow.

lordirah commented 1 year ago

I can help in maintaining it

DustinMoriarty commented 1 year ago

@codeforkjeff : Are you still looking for a maintainer? I would be happy to help.

codeforkjeff commented 7 months ago

@lordirah @DustinMoriarty Terribly sorry for ignoring this for so long. Might either of you be interested in taking over this repository as the primary maintainer? Not sure how best to do that (is it possible to transfer ownership of repositories?).

Also tagging @tom-juntunen: I see you recently forked this repo and reviewed a MR, do you have any interest in this role?

This repo is waaay behind the latest version of dbt at this point and probably needs a lot of work. 😞

tom-juntunen commented 7 months ago

Hi Jeff,I am interested, let me take a closer look and get back to you with my evaluation of what is needed. If you have a list started please share and we can discuss. Thanks,TomTom JuntunenOn Apr 18, 2024, at 9:22 AM, Jeff Chiu @.***> wrote: @lordirah @DustinMoriarty Terribly sorry for ignoring this for so long. Might either of you be interested in taking over this repository as the primary maintainer? Not sure how best to do that (is it possible to transfer ownership of repositories?). Also tagging @tom-juntunen: I see you recently forked this repo and reviewed a MR, do you have any interest in this role? This repo is waaay behind the latest version of dbt at this point and probably needs a lot of work. 😞

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

codeforkjeff commented 7 months ago

@tom-juntunen Much appreciated! I'm always surprised by how much interest and use dbt-sqlite gets, so you would be helping out a lot of people.

Let's use this github issue to continue discussing it. If you have any private concerns, you can email me directly, My address is on my website (listed on my profile).

tom-juntunen commented 7 months ago

Hi @codeforkjeff I have done the following work today:

Evaluate current state:

Run current tests:

Looked into v1.5 failures, which all tied back to the missing self.host attribute on the SQLiteCredentials object.

Learned of an open PR for v1.5 compatibility, which is a bit ahead of where I'm at considering it has picked up these items already:

Since I didn't need this PR to pass the dbt-sqlite tests using dbt-core v1.5, there needs to be a new unit test added for the new model contracts feature that fails without this PR, and succeeds with it. Our version of sqlite3 does not support STRICT tables, so we did not support model contract enforcement with dbt v1.5 and left a stub instead.

Next steps

  1. Naively run tests for new dbt-version using prior dbt-sqlite version
  2. Identify any new warnings or errors and find the root causes
  3. Review dbt release notes and commit history for files of interest to contextualize the root cause further
  4. Come up with a potential solutions for fixes and review PRs with other contributors
  5. Add any missing unit tests based on new features in dbt-core
  6. Ensure all tests run successfully
  7. Prepare for deployment and publishing an updated release

Also I will go through the Issues list and see if there is more context I can pick up there about the current state.

I think if we can get dbt-sqlite upted to v1.5.x before 4/27 that would be good progress.

codeforkjeff commented 7 months ago

Thanks for all this work so far! You're super organized. This big picture view is extremely valuable.

I would say that #51 with your suggested adjustment is good enough for cutting a 1.5.0 release. It's not worth any more effort after that, since dbt-core 1.5 is so close to end of life. If users report bugs with 1.5.0, we can evaluate from there and decide on further bug fix releases for 1.5.x.

After that, it might be best to jump straight to 1.7. It will still be important to look at the release notes for 1.6 to see what has cumulatively changed, but skipping a release/maintenance for 1.6 could save time and work on something whose end of life date is pretty soon, Jul 30th. I'm happy to leave it up to you to decide what's best.

I've sent an invite making you a maintainer of this repo. Feel free to branch, commit, merge, tag releases. We can coordinate uploading packages to PyPI when it comes time to do that.

I'll do my best to advise and keep in touch as much as I can, but I can't contribute too much beyond that right now, unfortunately.

tom-juntunen commented 7 months ago

I would say that https://github.com/codeforkjeff/dbt-sqlite/pull/51 with your suggested adjustment is good enough for cutting a 1.5.0 release. It's not worth any more effort after that, since dbt-core 1.5 is so close to end of life. If users report bugs with 1.5.0, we can evaluate from there and decide on further bug fix releases for 1.5.x.

Sounds good. I have pinged Matthew for making that final adjustment so we can have the 1.5.0 release ready for next week.

After that, it might be best to jump straight to 1.7. It will still be important to look at the release notes for 1.6 to see what has cumulatively changed, but skipping a release/maintenance for 1.6 could save time and work on something whose end of life date is pretty soon, Jul 30th. I'm happy to leave it up to you to decide what's best.

Sure thing. I will investigate the release notes and upgrade notes for maintainers for 1.6 and 1.7 listed here:

And if I have any questions will use existing issues here:

Thanks @codeforkjeff for your helpful comments and advice!

DustinMoriarty commented 6 months ago

@tom-juntunen: You appear to be very organized. That is great! I am still happy to pitch in to help @codeforkjeff and @tom-juntunen .

I have actually been pretty surprised that DBT Labs has not prioritized keeping one in-memory solution working well. I would think that they would swoop in to help with a project like this. It appears that they do a lot of their local testing with postgresql, which I get for an e2e level test, though I would think that one would always start with an in-memory solution that can be set up entirely in python for unit tests. I saw duckdb in some of the DBT Labs docs, which makes me think that they are using duckdb instead for quick in-memory use cases, which also seems like an odd choice compared to something as well supported and with as much history as sqlite.