Flexible switching for multihost/sharding support

[mikecmpbll]

Flexible switching

This will make multi-host support a first-class citizen in Apartment. By flexibly choosing the switching technique we can support tenants across many hosts. This makes logical sharding at the application level very simple.

• [x] If config.always_switch_connections = true (or similar), always establish a new connection on switch.
• [x] If the database host matches the current tenant's host, use 'local' switching (schemas on pg, use on mysql).
• [x] If the host does not match the current tenant's host, connect to the correct host first, and then do local swiching.
• [ ] Mirror connection_handler.establish_connection's API by accepting a URL, a configuration hash, or a database.yml connection symbol.
• [x] All modes of switching 100% threadsafe.

This will target Rails 5.1 and above only (due to the vastly improved connection handler API) and will not be backported.

[mikecmpbll]

all my work on this is here https://github.com/mikecmpbll/apartment/tree/flexible-switching. as far as I can tell it's pretty much ready, tests are all passing (new minitests :D), although not yet as comprehensive as I'd like

would be great if some people can test in their staging environments (with care!). we've not managed to upgrade our app to 5.1 yet that's currently in the works, so untested in our staging env.

• main difference is the concept of "tenant resolvers". it converts a tenant name into a full spec. it ships with two resolvers, Database, and Schema, which as you might expect use the tenant name to change the database or the schema parts of the database connection spec.

  require 'apartment/resolvers/database'

  Apartment.configure do |config| config.tenant_resolver = Apartment::Resolvers::Database end

• the resolved spec gets compared to the current spec and the switching strategy is worked out at runtime. this also works for specs with different hosts. you can also pass a full database connection spec (hash) to the switch method, and write your own resolvers.
• all excluded models use the same connection pool.

• config.prepend_environment etc is replaced with config.tenant_decorator (callable), e.g.

  Apartment.configure do |config| config.tenantdecorator = ->(tenant){ Rails.env.production? ? tenant : "#{Rails.env}#{tenant}" } end

  note that the result of the decorator is passed to the resolver.

[olivierlacan]

Thank you so much for working on this @mikecmpbll. This is one of the worries I have with starting to use this gem for a project that may or may not need to scale in the future.

[mikecmpbll]

for an example of how you might do sharding with the tenant resolver, we're experimenting with something like this:

require 'apartment/resolvers/abstract'

class CustomResolver < Apartment::Resolvers::Abstract
  NODES = %w(
    db1.myapp.net
    db2.myapp.net
    db3.myapp.net
    db4.myapp.net
  )

  def initialize(*)
    super
    @nodes = Clandestined::RendezvousHash.new(NODES)
  end

  def resolve(tenant)
    init_config.dup.tap do |c|
      c[:database] = tenant
      c[:host]     = @nodes.find_node(tenant)
    end
  end
end

Apartment.configure do |config|
  config.tenant_resolver = CustomResolver
end

this uses clandestined, a ruby gem for rendezvous hashing.

i intend to make some changes around connection management, but testing so far has proved successful on mysql.

[aldrinmartoq]

I have multiple tenants, all share the same schema except for one table which has different columns for every tenant. I'm actually doing Model.reset_column_information, but that doesn't scale.

Do you know if this new Database/Schema switcher can mantain a different column cache for each tenant?

[mikecmpbll]

@aldrinmartoq nope, not designed for that case i'm afraid.

[aldrinmartoq]

@mikecmpbll Right, thank you for your response.

I've ended reading AR code. My current solution overrides columns, column_names, … and a bunch of other AR methods to support my case, and it seems to work well.

[ryanswood]

@mikecmpbll Thank you for taking the initiative to rewrite this gem to support DB and schema switching. I work for Able Health and we greatly rely on this gem. We would love to contribute to finish this work as it perfectly matches our scaling plan. What needs to be done at this point and how can we help? I see above that you asked for help by testing on staging.

FYI, we are using Rails 5.1 and Postgresql.

[ryanswood]

@mikecmpbll What is the best way to get involved with the "Flexible switching" project and see how I can best contribute?

cc @bradrobertson

[mikecmpbll]

hi ryan. somewhat frustratingly (it's becoming a theme on this issue), i've been flat out on other things of late. the current state of play here is that we've done some minimal testing on our rails 5 fork of our application using mysql with this branch, and we've not seen any problems.

if you want to test it out, please feel free. i'm not sure what set up you intend to use but if it's just pg with schemas on a single host, you should be able to get up and running with the config.tenant_resolver = Apartment::Resolvers::Schema setting.

with that resolver selected, the strings passed to Apartment::Tenant.switch will be interpreted as schema names.

[nlsrchtr]

@mikecmpbll Thanks for this rewrite and the effort you put into this! I tried out this branch and it works pretty stable in combination with Postgresql. The one issue I found is, that when tenant_names is defined as a Hash:

config.tenant_names = lambda do
  Installation.all.each_with_object({}) do |installation, hash|
    hash[installation.tenant_name] = installation.database_configuration
  end
end

the tasks rake db:migrate and rake db:rollback are failing, because:

def tenants
  ENV['DB'] ? ENV['DB'].split(',').map { |s| s.strip } : Apartment.tenant_names || []
end

would return the hash, so I changed it into:

def tenants
  ENV['DB'] ? ENV['DB'].split(',').map { |s| s.strip } : Apartment.tenant_names.keys || []
end

But this will obviously not work for non-hash definitions of tenant_names in the config. Maybe you have a better fix, which would work in general.

[derosm2]

Any update for merging this PR? Would love to help out where needed.

[muyiwaoyeniyi]

@derosm2 I was trying to use your repo here - https://github.com/derosm2/apartment/tree/flexible-switching-rails-5-2 - but my rails server wouldn't start after installing the gem. Is there anything I need to know to install this gem?

[mikecmpbll]

@muyiwaoyeniyi might wanna give me a clue as to why it doesn't start? :)) everything of note is documented in this issue afaik.

[derosm2]

@muyiwaoyeniyi Sorry, we ended up moving to a different repo: https://github.com/PatientWisdom/apartment/commits/flexible-switching-rails-5-2

What is the issue?

[muyiwaoyeniyi]

@mikecmpbll I was referring to a fork of your repo by @derosm2 but I'm about to test your repo and will let you know if I run into any issue.

@derosm2 This is the issue I'm facing. I actually can't start the server or run apartment:install. See the error below.

Looks like it is looking for a default_tenant and maybe tied to this comment? - https://github.com/derosm2/apartment/pull/1#pullrequestreview-216714813

[derosm2]

@muyiwaoyeniyi This probably isn't the right place to discuss an issue with a separate fork, but feel free to comment over there with some more details (like your config).

[muyiwaoyeniyi]

@derosm2 I will do that. Thanks.