Single Tenant's Column Metadata Cached & Used for All Tenants

[sshaw]

Steps to reproduce

Add a column to a table table in only one schema:

alter table "some-schema".bars add column foo varchar(100);

Apartment::Tenant.switch("some-schema") do
  # Something to cause AR to generate a full list of column names for bars
  Bar.incudes(:relation).where(:relations => { :id => 9999999 }).to_a 
end

Apartment::Tenant.switch("schema-without-bars.foo") do
  # Will try to select bars.foo even though it does not exist
  # Will work if you run Bar.reset_column_information first
  Bar.incudes(:relation).where(:relations => { :id => 9999999 }).to_a 
end

Expected behavior

Columns that don't exist in the schema are not queried.

Actual behavior

We see this in production, we'll switch to a schema that does not have foo and run a query where AR generates the column list. This list includes foo and an ActiveRecord::StatementInvalid error is raised.

This is using Puma. Is this library thread-safe? Does not appear conclusive from all the open and closed issues.

System configuration

• Database: PostgreSQL 11.5 (pg 1.0.0)

• Apartment version: 2.2.0

• Apartment config (in config/initializers/apartment.rb or so):

require 'apartment/elevators/subdomain'

config.excluded_models = %w{ Foo }
config.tenant_names = lambda { Foo.pluck :name }
config.use_schemas = true

• Rails (or ActiveRecord) version: 5.2.3

• Ruby version: 2.5.7

[rthbound]

I believe we have seen the same problem. This from a teammate:

I was (sort of) able to reproduce the mysql error: First add a random column to some tenant DB in the mysql console:

\u 01DQN0P0JTDZEVPGPVTSF5SCWK

alter table parties add foo tinyint;

Then run the following from the rails console:

Tenant.switch_to_default
t1 = Tenant.find(1)
t1.switch_to_tenant_database

Party.first Cron::AppointmentReminderCheck.new.send(:find_sms_reminders_instances!)

t2 = Tenant.find(2) # some other tenant t2.switch_to_tenant_database

Party.first Cron::AppointmentReminderCheck.new.send(:find_sms_reminders_instances!)

=> ActiveRecord::StatementInvalid: Mysql2::Error: Unknown column 'parties.foo' in 'field list': SELECT

When we have lots of migrations, or a few long running migrations, then the probability of this occurring in production is highest after we finish a migration on the first tenant. The probability decreases as we migrate each of the remaining tenants.

The probability would be lowest if Apartment were able to migrate as follows:
- Run first migration against each tenant
- Run second migration against each tenant
- Etc.

Currently Apartment works as follows (highest probability of this error happening in production):
- Run first and second migration against first tenant
- Run first and second migration against second tenant
- Run first and second migration against third tenant
- Etc.

For now our best work-around is to `scp` migrations to a production server in advance of a deployment and run each migration explicitly:

rake db:migrate:up VERSION=20210101000000 rake db:migrate:up VERSION=20210101000001 rake db:migrate:up VERSION=20210101000002 rake db:migrate:up VERSION=20210101000003