Is this gem maintained?

[bloudermilk]

Our team is evaluating some options to read from our follower DBs on Heroku for select analytics workloads. We found this gem and while it seems to handle our use case, we had some issues in practice, which led us to the Issues tab.

It seems that issues and PRs are piling up over the last few of months, leaving us worried that we're adopting a library that might no longer be maintained. Of course we're very grateful to the OSS authors who shared this library with the public, but practically speaking it wouldn't be responsible for us to use this gem if it's been abandoned.

So I must ask, without any judgement, do the authors intend to continue maintaining this library?

[thiagopradi]

hey @bloudermilk - I try to do my best to work on the open issues / requests, but since I'm not using the library on a production anymore, It's been pretty hard to allocate time to work on it.

That said, there are a few users who are using the library successufully in their production environments. I would recommend you to try it out for a few days and see if it fits your use case.

Hopefully my comment was helpful.

Thiago

[pboling]

@bloudermilk From the perspective of a taker:

1. Many people use this library, and, IMO, it needs to be passed to community maintenance.

2. I use one feature of this gem, to direct queries explicitly to a master or a follower, and for that it usually works. There are a number of quirks and edge cases, as you will see in /issues. But I use that one feature on an absolute monster of a system (intricately.com - billions of jobs per month, trillions of queries, terabytes of PG data, hundreds of millions of records), and it has worked without any significant issues (nothing that didn't have a simple workaround).

3. Using the gem is not transparent. Crumbs of this gem will snake through the code. In certain configurations, like my setup, you do have to explicitly set the current_shard on objects before saving them, when they originate on the follower, for example. So you do need to commit to it when you choose it.

4. Judging from what @thiagopradi himself has said on other threads, the internals of the gem are in need of refactoring to solve some of the trickier things.

5. It looks like Rails 5.2 support may be difficult, or may necessitate that refactoring. For this project to survive there needs to be a community effort. @thiagopradi have you considered creating a Github Org, or adding other maintainers/owners to your repo?

6. I evaluated the landscape of tools like this for another project recently, and this project is still the best option, IMO, despite all the risks 🌟

This is a great discussion to have! I hope more people will jump in here, and I hope some of those will offer support. I just took over the oauth2 project which was in a similar problematic state of near-death, and don't have any extra bandwidth or I would volunteer.

[erkie]

@thiagopradi thanks for jumping in and responding here. I'm just curious how we can get Rails 5.2 support merged/worked on. The PR I made worked in production for us, but there are certain aspects of it that I think could be improved. So a discussion with people in-the-know would be great to have. (Please don't take it as me trying to pressure you or anyone to take a look at the PR! I'm just curious what the normal way of going about would be)

Are you the only maintainer on this repo with merge access? If not, how/where do you communicate/congregate? I definitely think a community org is a great idea to let this gem thrive!

[bloudermilk]

Thanks everyone for the responses so far–your promptness and thoughtfulness do well to answer my original question.

@thiagopradi I know first hand the reality of past OSS work becoming personally irrelevant and hard to dedicate time to as a result. I've had to leave projects behind in the past, but I wasn't nearly as successful in creating something popular in the first place. I think the continued popularity of this gem speaks to its utility and opportunity to outlive its original purpose at your previous org. Perhaps the suggestions above are reasonable and there are a few people with sufficient knowledge of the codebase to help maintain/refactor it?

@pboling thank you for your perspective and vote of confidence. It sounds like we have very similar or identical use-cases, which is encouraging. We're still on Rails 5.1 so it seems there is hope that we can make it work. I will spend a few more hours this week tinkering to see if that's the case.

[JasonLunn]

How many open issues are there in total impacting Rails 5.x support? I can think of:

5.2:

477

489

5.1+:

452

5.0+:

478

467

Have I missed any?

[thiagopradi]

Hi @bloudermilk, @pboling, and @erkie - thanks for the insights and comments on this thread. I'm going to try to summarize my thoughts about the mentioned topics here:

1 - Regarding passing it along to community maintenance, I agree and I'm willing to do this. However, I've never found someone who could contribute regularly to the Gem, working on the code and answering the issues. If any of you guys want to be a maintainer, please ping me.

2 - Regarding the fact that the Gem is not transparent, I agree 100%. I coded this Gem almost eight years ago in the context of Rails 2.x - which was super coupled and pretty hard to extend - which lead to a lot of monkey patch on the internals of the Gem. The Gem does need a big refactor to be more reliable, mainly when using in big legacy projects.

3 - On the matter of the Rails 5.x support, I'll try to allocate some time to work on it this week. I'll try to merge all the fixes and get a release that is compatible with Rails 5.x.

I'll keep this thread open for discussing regarding moving the gem to community maintenance - if you guys have any tips on this subject, please let me know.

Thiago

[JasonLunn]

@thiagopradi - what's the state of your efforts? Is there anything that can be done to help?

[pboling]

@JasonLunn I think most critically we need people with available time to work on the issues noted here: https://github.com/thiagopradi/octopus/issues/490#issuecomment-383667221 (three comments above this one)

[JasonLunn]

I don't think any one wants to take on any of the above without hearing from @thiagopradi first so as not to create redundant effort.

[adomokos]

Would it simplify things if this gem was refactored to support only AR 5.x+ and released under 2.0 version with no backward compatibility? We use it in pure Ruby apps with only AR dependency in them.

The reason I write this is to decrease the scope of this proposed refactoring.

[JasonLunn]

I'd have no objection to any approach that didn't break JRuby support.

[pboling]

This gem is struggling for maintenance with the current support load, so I vote to reduce the scope of supported Rubies and Railses with the next major release... Keeping latest JRuby 9k and MRI Ruby stable at minimum.

[lunaru]

@pboling you mentioned that you evaluated several other alternatives. We're using this gem at reamaze.com for the exact same as what you've mentioned (sending reads to primary vs secondary databases). Has your evaluation of the landscape changed in the last half-year since your original comment? It's an interesting conundrum if the primary sharding/read-splitting gem in the Rails community is effectively on life-support.

[pboling]

@lunaru This is this the gem we are using on a hugely scaled application that is pushing the limits of PostgreSQL v9, having many tables with hundreds of millions of rows, supporting many thousands of connections, and simultaneous queries. I haven't re-evaluated the landscape at all since my comment above.

We are planning to migrate from PG 9.6 to 11 when it is released (soon!) and I expect to run into problems with this library as we will begin using the partitioning features of PG 11 as the next step in our search for ways to keep the data beast humming.

At that point I will either be forced to take an active interest in this project, or write/find an alternative.

UPDATE: We migrated to v10, and v11 without issues.

[pboling]

Many, if not all, of the features of Octopus may be part of Rails 6! https://github.com/rails/rails/pull/34052

I am not yet sure how we will bridge the gap from Rails 4.2 to Rails 6 yet though. This gem may still play a role in a migration that steps through Rails 5.2, if we can get a release together that supports 5.2 well.

[thiagopradi]

Hi @pboling - I actually plan to create a migration guide (for Rails 6) from Octopus to the multi db API that is going to be released with Rails. For Rails 5.2 - I'm doing what I can to work on an Octopus release for Rails 5 - but I hadn't much free time in the last month.

[lunaru]

@thiagopradi I want to thank you for your continued involvement and updating us with the plan. It would be perfect if we could have a 5.2 compatible octopus to tide us over for the migration to Rails 6.x.

I think the reality is that even with 6.x on the horizon it will be months if not years for folks to fully migrate over so a 5.2 compatible octopus is still in heavy demand and much appreciated by the community. We have a codebase we're trying to migrate to 5.2 so if you need any help with testing your 5.2 branch, I'd be happy to provide feedback and help where I can from our team.

[kevinjcoleman]

@thiagopradi not sure if you saw this PR https://github.com/thiagopradi/octopus/pull/507, but I solved some of the migration issues you were having on the previous branch. I know you were looking for help previously, so I'm willing to jump in where ever you need. I'm testing that branch I created today in our Rails 5.2 app.

[thiagopradi]

@kevinjcoleman I haven't seen yet - I'm going to review it this week. It seems pretty feature-complete for am Octopus release that supports Rails 5.2 👏 👏 . Let me know if your tests with your app are successful.

[thiagopradi]

@kevinjcoleman - can you run your tests directly from my branch (feature/updating-octopus-versions) ? Tests are all green - I just want to make sure it works with a real Rails 5.2 app. Thanks!

[kevinjcoleman]

@thiagopradi I've started testing. For the most part it works, however I'm running into an issue, I don't think this is rails 5.2 specific due to this issue https://github.com/thiagopradi/octopus/issues/421, where running rake db:migrate is attempting to run migrations on my replicated slave and therefore breaking the replication due to duplicate key entry constraints in the ar_internal_metadata. It looks like those migrations are being run because connection.shards is being passed here rather than filtering by which shards are for that migration. https://github.com/thiagopradi/octopus/blob/feature/updating-octopus-versions/lib/octopus/migration.rb#L113

[kevinjcoleman]

@thiagopradi ok the migration issues I was having are solved by this PR https://github.com/thiagopradi/octopus/pull/508. If I'm understanding the way using is setup for migrations, we should only be passing the shards for a given migration like that.

[thiagopradi]

Hello everyone - an important announcement about Octopus: it will enter into maintenance mode once Rails 6 is released. Rails 6 will have all the features necessary for sharding built-in into Rails core: https://github.com/rails/rails/pull/34052. I'll provide a migration guide for everyone that is using Octopus to upgrade to the native AR API.

[krzcho]

@thiagopradi any chance on migration guide as we have rails 6 beta available?

[thiagopradi]

@krzcho - I'll try to get it done for the Rails 6.0 release in April.

[dhempy]

I am implementing Octopus in our Rails 4.2 application for reading from a replica. Default is fully_replicated: false, and we're only .using(:reader_db) for a handful heavy queries for reports.

Simultaneously, another team member is working on upgrading our app from Rails 4.2 to Rails 5.

I only just now, reading this thread, considered that Octopus may have issues under Rails 5. Can anyone advise if that's a turnkey/easy/painful transition from Rails 4.2?

[kevinjcoleman]

@dhempy honestly if you can I would wait until you get on Rails 6.1, it will have support for multiple databases and it will be under more consistent development than octopus. Doing that would be easier than trying to migrate from octopus to rails multiple databases at a later point which is a task I'm already dreading. https://edgeguides.rubyonrails.org/active_record_multiple_databases.html

[mobilutz]

@thiagopradi Is there any update on the migration guide? Thanks for a reply.

[sdrioux]

Hi @pboling - I actually plan to create a migration guide (for Rails 6) from Octopus to the multi db API that is going to be released with Rails. For Rails 5.2 - I'm doing what I can to work on an Octopus release for Rails 5 - but I hadn't much free time in the last month.

@thiagopradi Its been a year since this comment. Can you please write a migration guide or consider adding contributors to this gem?

[tibbon]

We're going to need to migrate soon, and even if there's no formal guide... any experience or hints about other people's migration experiences would be appreciated.

[veerae]

Is there any migration guide available, we are migrating our application into rails6, I need migration guide. Please provide soon if any one have

[krzcho]

@veerae I have switched to octoball