Remove DB2 provider

[rouault]

Remove DB2 provider

Date 2021/01/10

Author Even Rouault (@rouault)

Version QGIS 3.18

Summary

Remove the DB2 provider from the code base

Rationale

QGIS just got a new proprietary-based provider with the HANA provider, so for balance, let's get rid of an existing one. The original committer of the DB2 provider hasn't been active for more than 2 years, and all activity in the provider since then has been, as far as I can see, forced maintenance, due to changes elsewhere in the code base.

https://github.com/qgis/QGIS/issues?q=is%3Aissue+db2 shows that only one ticket has been created specifically regarding the DB2 provider. This is either a sign it is of the highest quality, but more realisticly than hardly anybody is using it, otherwise there would be feature requests regarding it.

IBM isn't a sustaining member of QGIS.org, so it is hard to justify maintenance costs due to it.

Backwards Compatibility

Breaks users of the functionality

Issue Tracking ID(s)

https://github.com/qgis/QGIS/pull/40930

Votes

(required)

[rouault]

Message advertizing this proposal on qgis-user: https://lists.osgeo.org/pipermail/qgis-user/2021-January/047690.html

[nyalldawson]

@mbernasocchi before we take this drastic step, so you think it's worthwhile to reach out to IBM (if possible!!) and see if they will step up to fund this maintenance?

[nyalldawson]

@dwadler as the original author of this provider do you have any comment here?

[nyalldawson]

(for reference I've also reached out to David via email to raise this proposal to his attention)

[dwadler]

I oppose the removal of the DB2 provider support. I am in the process of skills transfer to someone else who will be providing ongoing maintenance and enhancement support and I am still committed to the DB2 support in all of the open source geospatial projects which have this support.. We are looking at adding support for the DB2 Spatial Analytics which supports the column store of the DB2 warehouse.

It is difficult to know the level of usage of our spatial capabilities unless there are problems. The DB2 spatial support goes back to 2000 and since 2010 there have been very few customer problems.

[rouault]

First, I should underline that my following remarks aren't personal at all. I'm just acting as the bad cop. In 2 years, we'll probably have to do the same with the HANA provider.

who will be providing ongoing maintenance

The issue is more the cost of this provider (or any actually) to the rest of the contributors who want to do for example API changes and have to go through each provider. It is just a bit each time, but overall the costs add. We would probably have a different perception of the benefit vs cost ratio of the provider if its maintainers would participate to the general good of QGIS, and not only to the provider itself.

It is difficult to know the level of usage of our spatial capabilities

Hence we try to reach the QGIS community to know how many actually use that provider.

there have been very few customer problems.

I'm happy for your customers and I've no doubt that DB2 support in QGIS is good for you, but how good is that for QGIS ? How many QGIS users use the DB2 provider: < 0.1% ? The other issue I see is QGIS as a popular FOSS software doing free (gratis) advertising to a proprietary solution without getting any obvious return from that.

[dwadler]

I don't take this personally and understand your and QGIS's position.

It is still my plan to provide ongoing maintenance - if you need a name, associate mine. We also need to have a backup. Spatial technology has been a personal passion of mine for over 30 years.

Cost is a little more difficult to address. It is unlikely that IBM will provide funding directly to QGIS. I would be willing to be more involved in QGIS overall. I have two hats, one as an IBM employee until June 2021 and one as a personal spatial technologist.

Cost benefit is also difficult to ascertain. We have implemented DB2 spatial support across the main open source spatial projects - QGIS, GeoTools, GeoServer, uDig, GDAL, Hibernate Spatial. Individually it may not be a big benefit but illustrates a degree of commitment to the technology.

Similarly for QGIS, I would think that having a broad base of supporting providers would lend to its attractiveness, allowing users to move across providers in an agnostic fashion.

First, I should underline that my following remarks aren't personal at all. I'm just acting as the bad cop. In 2 years, we'll probably have to do the same with the HANA provider.

who will be providing ongoing maintenance

The issue is more the cost of this provider (or any actually) to the rest of the contributors who want to do for example API changes and have to go through each provider. It is just a bit each time, but overall the costs add. We would probably have a different perception of the benefit vs cost ratio of the provider if its maintainers would participate to the general good of QGIS, and not only to the provider itself.

It is difficult to know the level of usage of our spatial capabilities

Hence we try to reach the QGIS community to know how many actually use that provider.

there have been very few customer problems.

I'm happy for your customers and I've no doubt that DB2 support in QGIS is good for you, but how good is that for QGIS ? How many QGIS users use the DB2 provider: < 0.1% ? The other issue I see is QGIS as a popular FOSS software doing free (gratis) advertising to a proprietary solution without getting any obvious return from that.

[rouault]

GDAL

I also plan to retire it from GDAL: https://lists.osgeo.org/pipermail/gdal-dev/2021-January/053260.html

[mbernasocchi]

@mbernasocchi before we take this drastic step, so you think it's worthwhile to reach out to IBM (if possible!!) and see if they will step up to fund this maintenance?

@dwadler would it be possible for you to let me have (marco@qgis.org) an appropriate contact to try asking as QGIS project?

[dwadler]

GDAL

I also plan to retire it from GDAL: https://lists.osgeo.org/pipermail/gdal-dev/2021-January/053260.html

When you refer to "contributing to the overall good of the project as a reason to keep a driver", what would that entail?

[elpaso]

@dwadler I think this is about generic commitment to the project and its community, fixing bugs, helping out, documentation, refactoring, keeping up with new APIs etc..

For DB2 for instance, it doesn't implement the connections API (QgsAbstractDatabaseProviderConnection), that would be a nice start.

[rouault]

QGIS's position.

Just take this as my personal position. I'm not speaking on behalf of the project.

[nyalldawson]

Professional opinion:

We shouldn't remove this provider mid cycle, but instead should defer this decision till 4.0. I think the potential user harm of removing this while in the middle of a stable release is too high to give us this freedom.

Honest personal opinion:

As the one who originally reviewed and ultimately merged the db2 provider, I honestly have to admit that merging this work is one of the biggest regrets I have with respect to QGIS maintenance. While @dwadler was extremely responsive to changes during the review process, we have seen near-zero maintenance of this provider since, and it IS a burden which the whole (overworked) QGIS development community has inherited.

I was naive at the time, but I assumed that a responsive review cycle would follow through to a responsive code maintainer, not the "drop the code and run" approach which we got instead. I've been burnt, learnt my lesson, and would never do this again.

Now we have this massive provider which is a total unknown. I honestly suspect that it's broken in fundamental ways, and may not even be USABLE in ANY form on current QGIS versions. There's absolutely no way for QGIS developers to test this provider, and since it's initial drop we've seen massive changes in the QGIS codebase. The huge API changes with QGIS 3 and the external changes introduced by Qt 5 resulted in significant changes required to all the other providers, including the very widely tested postgres, ms sql and oracle providers. There were plenty of regressions in these providers in the early 3.x releases. This gives me zero confidence that the db2 provider (with its completely unknown user base) is an exception and magically kept working without issue. Rather, I strongly suspect it broke during this time and has remained broken ever since...

So now we have this probably-significantly-broken provider which we can't do ANYTHING with, and we've had no assistance from the commercial entity behind this provider (IBM) and no assistance from the original developer. We're effectively being held hostage here and being forced to work for IBM for free! :angry: Sorry for the strong language, but FUCK that.

(As it stands I'd far rather just delete the whole provider just because @rouault has issues with it, and because Even alone has done orders of magnitude more for the ENTIRE WORLD OF GEOSPATIAL USERS then IBM and their billions have EVER done. And to me protecting @rouault and his place in the foss4g community is so much more important then every single db2 user that exists.)

....phew.... deep breathes...

@dwadler please don't take any of this as a personal attack against you -- I can understand your viewpoint, and as a similar small business owner I understand why you aren't voluntarily maintaining this code without support from IBM. But PLEASE take the feedback from the project to your contacts at IBM and let them know the risk they are exposing their users to if they don't step up here and start financially supporting QGIS and GDAL and the rest of the open-source spatial ecosystem.

[rouault]

just because @rouault has issues with it

I'm only a marginal contributor to QGIS. So I don't really care if it gets removed (or moved to a plugin) now, in 4.0 or never. I'm not the one who will endure most pain due to its presence. I'm probably completely out of my role here, but I'm mostly sharing my experience with > 10 years of involvement with GDAL and seeing such pattern many times, and it seems that expressing my opinions in the form of a pull request at the distance of a click to the Merge button is a quite efficient way of triggering reactions :-) ... or not, because speaking about reactions: I know that qgis-user is followed by only a minority of users, but I should note that no one has expressed yet its need for the provider. If I had done the same with other proprietary-based providers, let's say the Oracle or MSSQL provider (their situation is different since they are actually supported by QGIS community developers), I'm pretty sure that QEP would be filled with tens of "no, you're crazy, don't do that". I'm not sure what being professional really means, but if we have no confidence the code works (I naively assumed it would work!), I don't really see why 3.18 wouldn't be a good target to remove it. What's the point of having broken code that no-one uses...

[dwadler]

Professional opinion:

We shouldn't remove this provider mid cycle, but instead should defer this decision till 4.0. I think the potential user harm of removing this while in the middle of a stable release is too high to give us this freedom.

Honest personal opinion:

As the one who originally reviewed and ultimately merged the db2 provider, I honestly have to admit that merging this work is one of the biggest regrets I have with respect to QGIS maintenance. While @dwadler was extremely responsive to changes during the review process, we have seen near-zero maintenance of this provider since, and it IS a burden which the whole (overworked) QGIS development community has inherited.

I was naive at the time, but I assumed that a responsive review cycle would follow through to a responsive code maintainer, not the "drop the code and run" approach which we got instead. I've been burnt, learnt my lesson, and would never do this again.

Now we have this massive provider which is a total unknown. I honestly suspect that it's broken in fundamental ways, and may not even be USABLE in ANY form on current QGIS versions. There's absolutely no way for QGIS developers to test this provider, and since it's initial drop we've seen massive changes in the QGIS codebase. The huge API changes with QGIS 3 and the external changes introduced by Qt 5 resulted in significant changes required to all the other providers, including the very widely tested postgres, ms sql and oracle providers. There were plenty of regressions in these providers in the early 3.x releases. This gives me zero confidence that the db2 provider (with its completely unknown user base) is an exception and magically kept working without issue. Rather, I strongly suspect it broke during this time and has remained broken ever since...

So now we have this probably-significantly-broken provider which we can't do ANYTHING with, and we've had no assistance from the commercial entity behind this provider (IBM) and no assistance from the original developer. We're effectively being held hostage here and being forced to work for IBM for free! 😠 Sorry for the strong language, but FUCK that.

(As it stands I'd far rather just delete the whole provider just because @rouault has issues with it, and because Even alone has done orders of magnitude more for the ENTIRE WORLD OF GEOSPATIAL USERS then IBM and their billions have EVER done. And to me protecting @rouault and his place in the foss4g community is so much more important then every single db2 user that exists.)

....phew.... deep breathes...

@dwadler please don't take any of this as a personal attack against you -- I can understand your viewpoint, and as a similar small business owner I understand why you aren't voluntarily maintaining this code without support from IBM. But PLEASE take the feedback from the project to your contacts at IBM and let them know the risk they are exposing their users to if they don't step up here and start financially supporting QGIS and GDAL and the rest of the open-source spatial ecosystem.

I'm not taking this personally. I do test the DB2 support with every QGIS release that comes out to verify that it is not broken although this testing should probably be done more regularly during the release cycle as it is too late to fix something by the time of the official release.

The reason that I do this work with a personal / business e-mail is so that I can still be involved without an IBM relationship and I do support the various DB2 drivers voluntarily. I retired in 2013 and came back in 2015 working for IBM one day a week as a retiree supplemental.

I have communicated the issue back to IBM but to be frank, in a huge organization like IBM it is difficult to arrange funding for projects that are not on the top of the strategic list like AI and cloud computing.

@nyalldawson I appreciate the support that you provided when we were developing the DB2 driver initially.

[gioman]

@nyalldawson @rouault thanks for taking this stance, hopefully this kind of messages will be heard loud and clear by this big players.

[mbernasocchi]

@dwadler

@mbernasocchi before we take this drastic step, so you think it's worthwhile to reach out to IBM (if possible!!) and see if they will step up to fund this maintenance?

@dwadler would it be possible for you to let me have (marco@qgis.org) an appropriate contact to try asking as QGIS project?

@dwadler ^

[dwadler]

@dwadler

@mbernasocchi before we take this drastic step, so you think it's worthwhile to reach out to IBM (if possible!!) and see if they will step up to fund this maintenance?

@dwadler would it be possible for you to let me have (marco@qgis.org) an appropriate contact to try asking as QGIS project?

@dwadler ^

I will get back to you next week.

[dwadler]

@dwadler

@mbernasocchi before we take this drastic step, so you think it's worthwhile to reach out to IBM (if possible!!) and see if they will step up to fund this maintenance?

@dwadler would it be possible for you to let me have (marco@qgis.org) an appropriate contact to try asking as QGIS project?

@dwadler ^

I've checked with our offering team and it just isn't going to be possible for IBM to provide financial support for QGIS at time. Sorry I don't have better news.

[rouault]

I've checked with our offering team and it just isn't going to be possible for IBM to provide financial support for QGIS at time. Sorry I don't have better news.

Thanks for the prompt action and the clear answer. At least we know where we are.

[rouault]

The DB2 provider will be hidden behind a configuration setting in 3.18 : https://github.com/qgis/QGIS/pull/41178

[mbernasocchi]

@dwadler

@mbernasocchi before we take this drastic step, so you think it's worthwhile to reach out to IBM (if possible!!) and see if they will step up to fund this maintenance?

@dwadler would it be possible for you to let me have (marco@qgis.org) an appropriate contact to try asking as QGIS project?

@dwadler ^

I've checked with our offering team and it just isn't going to be possible for IBM to provide financial support for QGIS at time. Sorry I don't have better news.

Thanks for the feedback