search

dpn-admin / dpn-server

Implementation of the DPN RESTful API.
Other
3 stars 6 forks source link

Cannot run script/setup_cluster.rb with default Gemfile #133

Open malakai97 opened 8 years ago

malakai97 commented 8 years ago

bundle exec script/setup_cluster.rb produces the error:

Couldn't create database for {"adapter"=>"sqlite3", "encoding"=>"utf8", "pool"=>5, "timeout"=>5000, "reconnect"=>true, "port"=>3306, "database"=>"db/impersonate_hathi.sqlite3"}
rake aborted!
Gem::LoadError: Specified 'sqlite3' for database adapter, but the gem is not loaded. Add `gem 'sqlite3'` to your Gemfile (and ensure its version is at the minimum required by ActiveRecord).

This is due to the file setting the environment variable to something not known to the Gemfile/bundler before loading the bundle, so only those dependencies not restricted to :development, :test, etc. will be installed and available.

Workaround for now is to add gem 'sqlite3' to your Gemfile.local.

To recreate this bug

On a system with no system-wide sqlite3 gem install, i.e.:

$ gem list

*** LOCAL GEMS ***

bigdecimal (1.2.6)
bundler (1.13.6)
coderay (1.1.1)
io-console (0.4.3)
json (1.8.1)
method_source (0.8.2)
minitest (5.4.3)
power_assert (0.2.2)
pry (0.10.4)
psych (2.0.8)
rake (10.4.2)
rdoc (4.2.0)
slop (3.6.0)
test-unit (3.0.8)

Run:

git clone git@github.com:dpn-admin/dpn-server.git
cd dpn-server/
rbenv local 2.3.1 
git checkout develop
bundle install --path .bundle
bundle exec rake config
bundle exec ./script/setup_cluster.rb  # this generates the error
malakai97 commented 8 years ago

Most likely applies to script/migrate_cluster.rb as well.

dazza-codes commented 8 years ago

Can't replicate this bug on a clean clone of the repository, using the following:

git clone git@github.com:dpn-admin/dpn-server.git
cd dpn-server/
rbenv local 2.3.1 
git checkout develop
bundle install
bundle exec rake config
bundle exec rake db:create  # might not be necessary
bundle exec rake db:migrate  # might not be necessary
bundle exec ./script/setup_cluster.rb 
bundle exec ./script/run_cluster.rb -f
malakai97 commented 8 years ago

Added better steps to recreate to the opening post with your snippet. It appears the script is picking up your system sqlite3 gem.

dazza-codes commented 8 years ago

Is this solved by moving the sqlite gem out of develop/test into the general env?

dazza-codes commented 8 years ago

I still can't replicate the problem using the develop branch, even after removing sqlite as follows:

$ gem clean
$ gem remove sqlite3
$ gem which sqlite3
ERROR:  Can't find ruby library file or shared library sqlite3
$ bundle show --paths | grep sqlite
/data/src/dlss/dpn/dpn-server/.bundle/ruby/2.3.0/gems/sqlite3-1.3.11
$ rm db/impersonate_*.sqlite3
# etc. to setup and run the cluster
dazza-codes commented 8 years ago

Note that the setup_cluster script creates the impersonate ENV when it does:

`cp config/environments/development.rb config/environments/impersonate_#{node}.rb`

Check that those files exist?

dazza-codes commented 8 years ago

Also, the bundler groups are not equivalent to any RAILS_ENV. Those groups are only used when doing installations and deployments. When the default bundle install is run, it should be installing all the gems (I think), unless you somehow have a config setting that is automatically excluding some groups? It's only a convention that these bundle group names are using the same names commonly used for RAILS_ENV.

For reference, see http://bundler.io/groups.html

dazza-codes commented 8 years ago

Check your config file, i.e.

$ cat .bundle/config 
---
BUNDLE_PATH: ".bundle"
BUNDLE_DISABLE_SHARED_GEMS: true
BUNDLE_BIN: ".binstubs"

If I run bundle install --without development, then the config looks like this:

$ cat .bundle/config 
---
BUNDLE_PATH: ".bundle"
BUNDLE_DISABLE_SHARED_GEMS: true
BUNDLE_BIN: ".binstubs"
BUNDLE_WITHOUT: development
malakai97 commented 8 years ago

bundler is the command line utility that performs the installation. Bundler is the ruby-facing portion of the gem that can be used to require all or part of the installed (bundled) gems. They're different things, so merely installing a gem into the bundle does not automatically require it in your application. The bundler documentation is quite accessible if you want to know more about how it works and what it does/doesn't do.

Rails requires the bundle via Bundler.require(*Rails.groups), which is informed by the RAILS_ENV variable. The gems in the :development group are not loaded when Rails.groups doesn't resolve to something that includes it, as is the case when we set RAILS_ENV=impersonate_sdr.

Simply moving the sqlite3 gem declaration in the Gemfile from the development group to the default group is not a good solution, because it now states that a gem that dpn-server doesn't actually depend on is one of its dependencies. When I mentioned adding sqlite3 to the Gemfile.local (and thus the default group), I intended it only as a workaround for this issue. It hides the problem, but the problem is still there.

I do not see an obvious fix for this, as it is a result of spoofing multiple different instances by manipulating the environment. This manipulation tends to violate the assumptions Rails makes, and this is another symptom.

dazza-codes commented 8 years ago

Just to be double-sure, I re-ran the commands in the updated description on a clean clone and still can't replicate the problem.

dazza-codes commented 8 years ago

Yes, I understand bundler as the installation vs. Bundle to require a set of gems. They can be orthogonal groups. I thought you might be having installation problems.

With regard to the require problem, here's what I find:

To experiment, I commented out the require 'bundler/setup' in boot.rb and added a puts to report the Rails.groups and, even so, the server starts up:

$ RAILS_ENV=impersonate_sdr bundle exec rails s
[:default, "impersonate_sdr"]
=> Booting WEBrick
=> Rails 4.2.7.1 application starting in impersonate_sdr on http://localhost:3000
=> Run `rails server -h` for more startup options
=> Ctrl-C to shutdown server
[2016-11-02 15:02:31] INFO  WEBrick 1.3.1
[2016-11-02 15:02:31] INFO  ruby 2.3.1 (2016-04-26) [x86_64-linux]
[2016-11-02 15:02:31] INFO  WEBrick::HTTPServer#start: pid=5724 port=3000

So, whatever the :default includes, it must have sqlite in it somehow? If I look at the http://localhost:3000/status/all.json, it contains a report on the db status that indicates it is OK.

Here's the details on the modifications to investigate the *Rails.groups:

$  git diff
diff --git a/config/application.rb b/config/application.rb
index e51f164..c91526f 100644
--- a/config/application.rb
+++ b/config/application.rb
@@ -10,6 +10,8 @@ require 'rails/all'

 # Require the gems listed in Gemfile, including any gems
 # you've limited to :test, :development, or :production.
+groups = *Rails.groups
+puts "#{groups}"
 Bundler.require(*Rails.groups)

 module DPN
diff --git a/config/boot.rb b/config/boot.rb
index 69d7ba4..a508f0e 100644
--- a/config/boot.rb
+++ b/config/boot.rb
@@ -5,4 +5,4 @@

 ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)

-require 'bundler/setup' # Set up gems listed in the Gemfile.
+#require 'bundler/setup' # Set up gems listed in the Gemfile.
malakai97 commented 8 years ago

I feel like this is entering twilight zone territory, because I'm seeing similar behavior. The fresh clone instructions still fail, but I'm trying to investigate each command individually but this doesn't run into the error.

Since it's so minor, I don't really want to spend more time on this now.

dazza-codes commented 8 years ago

I can sort-a replicate this problem using the branch gemset-crazy-loading, which uses Bundle.setup instead of Bundle.require. When it's done that way, your right, there is no bundle group for any of the RAILS_ENV=impersonate_* envs.

malakai97 commented 7 years ago

Changes in the aforementioned gemset-crazy-loading included here to remove some clutter elsewhere:

From 4cba2c5dc349d2b2683c6975502bafec3525307c Mon Sep 17 00:00:00 2001
From: "Darren L. Weber, Ph.D" <dweber.consulting@gmail.com>
Date: Wed, 2 Nov 2016 15:21:00 -0700
Subject: [PATCH] Gemset loading - use Bundle.setup instead of
 Bundle.require????

---
 config/application.rb | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/config/application.rb b/config/application.rb
index e51f164..cba7acd 100644
--- a/config/application.rb
+++ b/config/application.rb
@@ -4,13 +4,14 @@
 # See LICENSE.md for details.

-require File.expand_path('../boot', __FILE__)
-
-require 'rails/all'
+#require File.expand_path('../boot', __FILE__)

+require 'rubygems'
 # Require the gems listed in Gemfile, including any gems
 # you've limited to :test, :development, or :production.
-Bundler.require(*Rails.groups)
+require 'bundler'
+Bundler.setup(*Rails.groups)
+require 'rails/all'

 module DPN
   module Server
-- 
2.7.4