Docker automated build fails

[oveits]

At the end of the build log, I get:

Loading application environment...
Loading code in search of Active Record models...
Generating Entity-Relationship Diagram for 9 models...
Removing intermediate container 2d28febf271e
The command '/bin/sh -c bundle install; bundle exec rake db:migrate RAILS_ENV=development' returned a non-zero code: 1

and the build process is broken.

[oveits]

Test in the lab does not show the problem:

When performing a bundle exec rake db:migrate RAILS_ENV=development in the lab, I get:

provisioningengine@ProvisioningEngine:~/ProvisioningEnginev0.5.20_testfolder$ bundle exec rake db:migrate RAILS_ENV=development
Loading application environment...
Loading code in search of Active Record models...
Generating Entity-Relationship Diagram for 13 models...
Warning: Ignoring invalid association :resource on ActiveAdmin::Comment (polymorphic interface Resource does not exist)
Warning: Ignoring invalid association :author on ActiveAdmin::Comment (polymorphic interface Author does not exist)
Done! Saved diagram to erd.pdf.
provisioningengine@ProvisioningEngine:~/ProvisioningEnginev0.5.20_testfolder$ echo $?
0

I.e. here, I get a 0 success code instead of a 1 (failure).

[oveits]

It looks like the GraphViz gem is causing the error. In the Docker Hub log we find:

[91mrake aborted!
Saving diagram failed!
Verify that Graphviz is installed and in your path, or use filetype=dot.
/usr/local/bundle/gems/rails-erd-1.4.5/lib/rails_erd/diagram/graphviz.rb:207:in `rescue in block in <class:Graphviz>'
/usr/local/bundle/gems/rails-erd-1.4.5/lib/rails_erd/diagram/graphviz.rb:198:in `block in <class:Graphviz>'
/usr/local/bundle/gems/rails-erd-1.4.5/lib/rails_erd/diagram.rb:142:in `instance_eval'
...

but further below that the DB migration is still successful. In any case, it seems that the rake command fails (on Docker hub only, though! travisCI and Heroku have no problems, it seems).

[oveits]

Trying to ignore any rake failure as a test:

# Dockerfile
RUN bundle install; bundle exec rake db:migrate RAILS_ENV=development || echo "ignoring failure"

Success:

[oveits]

Now we need to find a better solution. The workaround above will ignore any rake failures, and this does not make sense.

I have tried out the hint on http://code.eklund.io/blog/2014/05/27/ignoring-heroku-errors-on-rake-db-migrate/:

config.active_record.schema_format = :ruby 

to config/environments/development.rb.

However, the erd.pdf file is still created locally:

$ bundle exec rake db:migrate RAILS_ENV=development
Loading application environment...
Loading code in search of Active Record models...
Generating Entity-Relationship Diagram for 13 models...
Warning: Ignoring invalid association :resource on ActiveAdmin::Comment (polymorphic interface Resource does not exist)
Warning: Ignoring invalid association :author on ActiveAdmin::Comment (polymorphic interface Author does not exist)
Done! Saved diagram to erd.pdf.

Also on Docker Hub, this did not work:

[oveits]

Note, that if it had worked, I needed to migrate to the production environment. See issue #49.

Maybe the workaround on https://github.com/voormedia/rails-erd/issues/70 works?

I have another idea: we can disable the gem in case of a production environment as a test, whether this can help, let us comment out the gem at all:

# Gemfile
# OV for automatic creation of UML-like class diagrams (see http://rails-erd.rubyforge.org/)
group :development do
  #gem "rails-erd"
end

Now it works:

[oveits]

I think, the best solution would be to disable the gem if and only if we are running the Dockerfile. My idea is like follows:

• Gemfile is just ruby code, so we can add any if clause around the gem command
• We could create an if clause like follows:

group :development do
  gem "rails-erd" if ENV["DOCKER"].nil?
end

In the Dockerfile, we just can set Docker to "true" (or any other value), and the gem should be skipped upon built. Let us try the following:

# Dockerfile
RUN export DOCKER=true; bundle install; bundle exec rake db:migrate RAILS_ENV=development

Bingo: that has worked!

[oveits]

The change has induced a problem with travisCI:

3.94s$ bundle install --jobs=3 --retry=3 --deployment
You are trying to install in deployment mode after changing
your Gemfile. Run `bundle install` elsewhere and add the
updated Gemfile.lock to version control.
You have added to the Gemfile:
* rails-erd

TODO: After performing a change, I need to run bundle install locally and add Gemfile.lock to the committed files.

Note: this is not a travisCI problem per se. We can see the same error message locally, if we run bundle install --deployment:

$ bundle install --deployment
You are trying to install in deployment mode after changing
your Gemfile. Run `bundle install` elsewhere and add the
updated Gemfile.lock to version control.

You have added to the Gemfile:
* rails-erd

[oveits]

Let us try to run bundle install before running bundle install --deployment on travisCI. For that we add the bundle install command on .travis.yml:

# .travis.yml
...
before_install:
  - bundle install

This seems to work:

Also Docker Hub automated build is working fine:

However, the additional bundle install has some drawbacks:

• bundle install is called twice, once with --deployment and once without. This is prolonging the build process by ~100%
• with this process, you cannot know, whether the gems and gem versions you use in the development machine are the same that are tested via travisCI

The latter disadvantage is prohibitive. If we test on travisCI, we want to have the same environment as the one we have locally and the one we have in the Docker image.

Summary: I have found a process that works locally, on travisCI and on Docker hub, but I am still not happy with it, since we could have different problems in different environments and we cannot easily test those environments in a different location (e.g. I would like to make sure a test that works locally also is successful on travisCI and is successful when using the Docker image).

TODO: find a better solution than the one with the additional bundle install on travisCI.

[oveits]

Brainstorming:

• local tests:
  • make sure you test on a clean local repository (unless this is only a transitory test for troubleshooting)
  • avoid gems that require additional SW installed (like rails-erb, which requires graphwiz). In the current case, I could create an extra environment graphviz that is built only, if a new UML graph is needed.
  • test with bundle install --deployment locally, if needed (e.g. if it fails on travisCI)
• travisCI:
  • test in different environments, not only development
• Docker Hub: how can we make sure that the image works fine? Can we test Docker images instead of or in addition of the SW itself?

Cool: https://docs.travis-ci.com/user/docker/ shows that we have many options on travisCI: 1) we can create a Docker image from Github, run, test it and push it to Docker Hub. 2) we also can pull a Docker image from Docker Hub and test it.

• problem with option 2): travisCI is triggered by a change on Github, not a new image on Docker Hub. At the time travisCI runs, the docker image will not be ready yet, since Docker Hub often takes some time until the build process is even started.
• another problem with option 2): travisCI will not detect a build problem on Docker Hub. It will just download the last successful docker image.
• topic with 1) and 2): we still might want to have quick tests based on the pushed source code, independent of the docker test (might be quicker).
  • different branches? For 2) we could use different repositories: one with the source code and one just with a .travis.yml to perform the testing. For 1), it is more likely that we use different branches. The source code is needed for creation of the docker image, so a separate repository with .travis.yml only does not make sense.
  • e.g. development and docker branch
• Can we do both, test the SW from source first (with immediate email sent) and then build and test the Docker image with a single .travis.yml file? I guess, no.

TODO: get acquainted with travisCI docker service

[oveits]

Workaround: 0) for the problem: comment out rails-erd in Gemfile, bundle install and commit the changes. At the same time, we can comment out the bundle install from .travis.yml.

For updating the UML PDF file: 1) uncomment rails-erd in Gemfile, 2) bundle install 3) perform the rake command 4) comment rails-erd in Gemfile again. 5) bundle install This way. Gemfile and Gemfile.lock do not get out of sync.

[oveits]

Problem with bundle install:

Oups, I have found that now, if I issue bundle or bundle install on my development machine, all gems are installed into vendor/bundle/. On http://bundler.io/bundle_install.html, this behavior is described for the command `bundle install --deploymentI had used recently to troubleshoot the travisCI issue. But now, it always does it even without the--deployment`` flag given.

Solution: to switch back, you need to call bundle install with the --system flag! After that, bundle install can be called without any flag and it will behave as it did before I had played around with the -deployment flag.

Cleaning rm -Rf vendor/bundle/ is cleaning the project folder from the gems I do not need there.

[oveits]

Now I have commented out the bundle installin travisCI and the rail-erd gem in the Gemfile and I have pushed the cahnage to Github.

Result both, travisCI and Docker Hub image creation are successful. We can close the issue, since I will keep the workaround. No messing around with different gems for different environments.