alison985
commented
1 year ago While Microsoft partnering with dbt labs is great, and they absolutely should be devoting staff time to dbt, this is potentially perilous for them and us(the community). Given a) existing infrastructure in Corporate America won't change to Microsoft Fabric from SQL Server anytime soon and b) Microsoft changes its mind about their data strategy frequently(e.g. 4 row chart above), what happens to existing Enterprise Microsoft customers now? What happens in the future when Microsoft changes its mind on data again?
JustGitting
cody-scott
mikaelene
ericmuijsvanoord
capitantoto
schlich
We recently got quite a few issues and comments asking for a status update on support for newer dbt versions, and I think I probably should have written this status update a while ago.
TL;DR we'll restructure some things, but it's going to be easier to support newer dbt versions :)
Thank you
Maybe I owe you an introduction. 🙃 I've been maintaining this adapter for the past couple of years and took it over a bit after the tremendous work by @dataders, @JCZuurmond , @mikaelene, @jacobm001, and ❤️ all other amazing contributors for which I'm forever grateful! I don't work for Microsoft, nor for dbt Labs. I do work with both companies as this adapter brings together the 2 things I love: dbt and the Microsoft stack.
Fabric: future for analytics in the Microsoft ecosystem
Unless you've lived under a rock 🪨😅, you might have seen Microsoft's announcements about Microsoft Fabric. This is the future for analytics in the Microsoft stack and we think this is probably where you should look first if you're looking for a good data warehouse to use with dbt. Note that I never said that other products or adapters will be discontinued. That is not the case. Now, who is this "we" I'm talking about?
Official Microsoft support
@dataders (dbt Labs), @prdpsvs (Microsoft), and myself have been exploring the options to bring dbt support to Fabric at launch and improve the support for dbt in the Microsoft stack in general. The result is there as dbt for Fabric is supported and maintained by Microsoft. The adapter for Synapse Dedicated SQL has also been transferred to Microsoft for support and maintenance.
This takes a big burden away from the community to provide all maintenance and support and since it is still fully open source, contributions are welcome just like before, so a bit of the best of both worlds. Where does that leave this adapter then?
Adapter inheritance
To explain that better, we need a bit more context:
This project, dbt-sqlserver, was the main adapter we were maintaining. dbt-synapse was a child adapter inheriting from this adapter where we only had to implement things for differences between SQL Server and Synapse Dedicated SQL. The experimental adapter for Synapse Serverless is another child adapter with dbt-synapse as its parent. With the introduction of Fabric and the news that Microsoft would be maintaining that adapter, we had to rethink this.
All source code from dbt-sqlserver has been transferred and refactored into the dbt-fabric adapter. That means Microsoft fully owns (as in: will maintain it) end-to-end. We can leverage this by shifting the inheritance in the adapters now. In the future, dbt-sqlserver, dbt-synapse, and dbt-synapse-serverless will become child adapters of dbt-fabric.
Community impact
This also means that a lot of source code in this repo is going to be removed as this has been incorporated into dbt-fabric and you'll mostly see the implementation for features different from Fabric here. I hope you understand that is a good thing 😀 As Microsoft can make dedicated time to maintain dbt-fabric, the community has to spend less time on maintaining this adapter so we can more quickly prepare support for newer dbt versions.
Where does that leave you? First, there is the refactoring work to switch those inheritances. Thanks to dbt Labs's great work on automated adapter testing, this should go smoothly. Next, we have to make sure all submitted PRs and issues get redirected to the right repo (either this one or dbt-fabric). I will do what I can there, but as always, contributions are welcome.
As with a lot of community projects, all of this work is not our day job and we're not getting paid for this. It all happens on a best-effort basis when the interested parties can dedicate some spare time to this project. I do believe the changes proposed above will make it a lot easier to maintain all of this since I'm personally more probably working with dbt-fabric in my day job than with the other adapters in the future and the goal is that adapter changes for the child adapters are limited to the absolute minimum.