fractaledmind / litestream-ruby

MIT License
71 stars 6 forks source link

New feature: Multiple database suport in Rails initializer #1

Closed dbackeus closed 4 months ago

dbackeus commented 8 months ago

Could be useful when partitioning across multiple databases.

fractaledmind commented 7 months ago

Yep, I needed to learn a couple new tricks to figure out how to even start on this, but I've learned them now:

https://fractaledmind.github.io/2024/01/15/rails-engine-migration-generator-with-support-for-multiple-databases/ https://fractaledmind.github.io/2024/01/02/sqlite-quick-tip-multiple-databases/

supermomonga commented 7 months ago

Hi, how about this interface?

# config/initializers/litestream.rb
Litestream.configure do |config|
  database_paths =
    ActiveRecord::Base.
    configurations.
    configs_for(env_name: Rails.env, include_hidden: true).
    map(&:database)

  database_paths.each do |database_path|
    replica1 = Litestream::Configuration::Replica.new(
      replica_url: 's3://as-you-wish1.localhost:9000/',
      replica_key_id: 'minioadmin',
      replica_access_key: 'minioadmin'
    )
    replica2 = Litestream::Configuration::Replica.new(
      replica_url: 's3://as-you-wish2.localhost:9000/',
      replica_key_id: 'minioadmin',
      replica_access_key: 'minioadmin'
    )

    database = Litestream::Configuration::Database.new(
      database_path:,
      replicas: [replica1, replica2]
    )

    config.databases.append(database)
  end
end

# lib/litestream.rb
module Litestream
  class << self
    attr_accessor :configuration
  end

  def self.configure
    self.configuration ||= Configuration::Root.new
    yield(configuration)
  end

  module Configuration
    class Root
      attr_accessor :databases

      def initialize(databases: [])
        @databases = databases
      end
    end

    class Database
      attr_accessor :database_path, :replicas

      def initialize(database_path:, replicas: [])
        @database_path = database_path
        @replicas = replicas
      end
    end

    class Replica
      attr_accessor :replica_url, :replica_key_id, :replica_access_key

      def initialize(replica_url:, replica_key_id:, replica_access_key:)
        @replica_url = replica_url
        @replica_key_id = replica_key_id
        @replica_access_key = replica_access_key
      end
    end
  end
end
fractaledmind commented 7 months ago

Hmm... My initial gut reaction is to prefer pushing developers to use the Litestream configuration file itself. My fuzzy plan for this was to make the install generator dynamically build the Litestream configuration file based on the databases listed in the database.yml. This feels like a decent compromise, because developers don't need to duplicate the database information from the database.yml into the config/litestream.yml file, but it also keeps the indirection/abstraction on top of Litestream minimal.

supermomonga commented 7 months ago

Am I correct in understanding that you mean, to provide a rule for an implicit correspondence between database.yml and environment variables?

For example, something like $LITESTREAM_DATABASE_PATH_[database_config_name].

# database.yml
development:
  web:
    database: 'db/development_web.sqlite3' # => will be assigned to $LITESTREAM_DATABASE_PATH_WEB
  litejob:
    database: 'db/development_litejob.sqlite3' # => will be assigned to $LITESTREAM_DATABASE_PATH_LITEJOB
production:
  web:
    database: 'db/production_web.sqlite3' # => will be assigned to $LITESTREAM_DATABASE_PATH_WEB
  litejob:
    database: 'db/production_litejob.sqlite3' # => will be assigned to $LITESTREAM_DATABASE_PATH_LITEJOB
fractaledmind commented 7 months ago

It wasn't my initial thought, no. I was simply thinking to make this generator template dynamic: https://github.com/fractaledmind/litestream-ruby/blob/main/lib/litestream/generators/litestream/templates/config.yml

When you run the install generator, the gem would introspect on your database configuration and setup the Litestream configuration to account for each SQLite database listed in your database.yaml.

I was also thinking that for the install generator, it would default to sending each database to the same bucket, just to different paths within that bucket.

supermomonga commented 7 months ago

Ah, that seems very smart approach for me.