Closed ShockwaveNN closed 6 years ago
I was suffered from this issue as well.
After hours of extensive research, it originates from an upstream issue of Bundler.
Please refer to bundler/bundler#6154 and bundler/bundler#6162.
===
The workaround provided by @lucasmazza is working.
You can add the environment variable BUNDLE_GEMFILE=./Gemfile
to revert to the original behaviour.
Should we introduce BUNDLE_GEMFILE=./Gemfile
into all images with bundler ~1.16.0?
Or should we just wait for a fix in the upstream bundler?
@gavinkflam thanks for mentioning this on this repository. Since we stumbled upon this on our app more and more I believe this should be fixed by removing the environment variables on the image rather than changing bundler - I believe that the image assumes too much of how the people are using the image and "installing" their apps with it.
@lucasmazza I totally agree with the image being too opinionated. I think the whole block I mention here could probably be removed except for the BUNDLE_SILENCE_ROOT_WARNING=1
, and use default bundler's values.
@yosifkit any thoughts on trimming down the environment variables on the images?
Current images with bundler 1.16.1 work fine with equivalent Dockerfile from above.
$ docker build .
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM ruby:2.4
---> dbe4f0ae7c72
Step 2/2 : RUN set -ex; git clone https://github.com/jwt/ruby-jwt.git; cd ruby-jwt; bundle install; cd ..; git clone https://github.com/janlelis/clipboard.git; cd clipboard; bundle install; cd ..; ls -la /clipboard/Gemfile.lock
---> Running in e5a0695ade91
+ git clone https://github.com/jwt/ruby-jwt.git
Cloning into 'ruby-jwt'...
+ cd ruby-jwt
+ bundle install
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Fetching rake 12.3.0
Installing rake 12.3.0
Using bundler 1.16.1
Fetching docile 1.1.5
Installing docile 1.1.5
Fetching json 2.1.0
Installing json 2.1.0 with native extensions
Fetching simplecov-html 0.10.2
Installing simplecov-html 0.10.2
Fetching simplecov 0.13.0
Installing simplecov 0.13.0
Fetching codacy-coverage 1.1.8
Installing codacy-coverage 1.1.8
Fetching codeclimate-test-reporter 1.0.8
Installing codeclimate-test-reporter 1.0.8
Fetching diff-lcs 1.3
Installing diff-lcs 1.3
Fetching ffi 1.9.22
Installing ffi 1.9.22 with native extensions
Using jwt 2.1.0 from source at `.`
Fetching rbnacl 5.0.0
Installing rbnacl 5.0.0
Fetching rspec-support 3.7.1
Installing rspec-support 3.7.1
Fetching rspec-core 3.7.1
Installing rspec-core 3.7.1
Fetching rspec-expectations 3.7.0
Installing rspec-expectations 3.7.0
Fetching rspec-mocks 3.7.0
Installing rspec-mocks 3.7.0
Fetching rspec 3.7.0
Installing rspec 3.7.0
Fetching simplecov-json 0.2
Installing simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
+ cd ..
+ git clone https://github.com/janlelis/clipboard.git
Cloning into 'clipboard'...
+ cd clipboard
+ bundle install
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Fetching rake 11.3.0
Installing rake 11.3.0
Using bundler 1.16.1
Using clipboard 1.1.1 from source at `.`
Using diff-lcs 1.3
Using rspec-support 3.7.1
Using rspec-core 3.7.1
Using rspec-expectations 3.7.0
Using rspec-mocks 3.7.0
Using rspec 3.7.0
Bundle complete! 4 Gemfile dependencies, 9 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
+ cd ..
+ ls -la /clipboard/Gemfile.lock
-rw-r--r-- 1 root root 661 Feb 23 20:23 /clipboard/Gemfile.lock
Removing intermediate container e5a0695ade91
---> 9d12bfabeb9b
Successfully built 9d12bfabeb9b
Can confirm that issue is not actual for your variant of Dockerfile:
FROM ruby:2.4
RUN set -ex;\
git clone https://github.com/jwt/ruby-jwt.git;\
cd ruby-jwt;\
bundle install;\
cd ..;\
git clone https://github.com/janlelis/clipboard.git;\
cd clipboard;\
bundle install;\
ls -la /clipboard/Gemfile.lock
But issue still actual for me with with my original Dockerfile from first message
FROM ruby:2.4
RUN git clone https://github.com/jwt/ruby-jwt.git
RUN cd ruby-jwt && bundle install
RUN git clone https://github.com/janlelis/clipboard.git
RUN cd clipboard && bundle install
CMD ls -la /clipboard/Gemfile.lock
I think issue should be reopened
I'm not able to reproduce with that exact Dockerfile
: :confused:
$ cat Dockerfile
FROM ruby:2.4
RUN git clone https://github.com/jwt/ruby-jwt.git
RUN cd ruby-jwt && bundle install
RUN git clone https://github.com/janlelis/clipboard.git
RUN cd clipboard && bundle install
CMD ls -la /clipboard/Gemfile.lock
$ docker build --pull .
Sending build context to Docker daemon 2.048kB
Step 1/6 : FROM ruby:2.4
2.4: Pulling from library/ruby
4176fe04cefe: Already exists
851356ecf618: Already exists
6115379c7b49: Already exists
aaf7d781d601: Already exists
acd6992c338a: Pull complete
91d9490effb0: Pull complete
1deabe842a08: Pull complete
Digest: sha256:eeb289458b9025d9c6c598739ca88e0e0e92e3fa28b8140b0e611f7363961eab
Status: Downloaded newer image for ruby:2.4
---> 4b7621478e79
Step 2/6 : RUN git clone https://github.com/jwt/ruby-jwt.git
---> Running in 06f72f0071b9
Cloning into 'ruby-jwt'...
Removing intermediate container 06f72f0071b9
---> c8a5341e65b9
Step 3/6 : RUN cd ruby-jwt && bundle install
---> Running in 4be1b26aa2b6
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Fetching rake 12.3.0
Installing rake 12.3.0
Using bundler 1.16.1
Fetching docile 1.1.5
Installing docile 1.1.5
Fetching json 2.1.0
Installing json 2.1.0 with native extensions
Fetching simplecov-html 0.10.2
Installing simplecov-html 0.10.2
Fetching simplecov 0.13.0
Installing simplecov 0.13.0
Fetching codacy-coverage 1.1.8
Installing codacy-coverage 1.1.8
Fetching codeclimate-test-reporter 1.0.8
Installing codeclimate-test-reporter 1.0.8
Fetching diff-lcs 1.3
Installing diff-lcs 1.3
Fetching ffi 1.9.23
Installing ffi 1.9.23 with native extensions
Using jwt 2.1.0 from source at `.`
Fetching rbnacl 5.0.0
Installing rbnacl 5.0.0
Fetching rspec-support 3.7.1
Installing rspec-support 3.7.1
Fetching rspec-core 3.7.1
Installing rspec-core 3.7.1
Fetching rspec-expectations 3.7.0
Installing rspec-expectations 3.7.0
Fetching rspec-mocks 3.7.0
Installing rspec-mocks 3.7.0
Fetching rspec 3.7.0
Installing rspec 3.7.0
Fetching simplecov-json 0.2
Installing simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container 4be1b26aa2b6
---> 9e59a5707974
Step 4/6 : RUN git clone https://github.com/janlelis/clipboard.git
---> Running in 7dbf43405646
Cloning into 'clipboard'...
Removing intermediate container 7dbf43405646
---> 83cc647e79c0
Step 5/6 : RUN cd clipboard && bundle install
---> Running in b3a1b93b38f2
Using rake 12.3.0
Using bundler 1.16.1
Using docile 1.1.5
Using json 2.1.0
Using simplecov-html 0.10.2
Using simplecov 0.13.0
Using codacy-coverage 1.1.8
Using codeclimate-test-reporter 1.0.8
Using diff-lcs 1.3
Using ffi 1.9.23
Using jwt 2.1.0 from source at `.`
Using rbnacl 5.0.0
Using rspec-support 3.7.1
Using rspec-core 3.7.1
Using rspec-expectations 3.7.0
Using rspec-mocks 3.7.0
Using rspec 3.7.0
Using simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container b3a1b93b38f2
---> 72c29d7ea676
Step 6/6 : CMD ls -la /clipboard/Gemfile.lock
---> Running in ebc57e5b2b3e
Removing intermediate container ebc57e5b2b3e
---> e9519654e875
Successfully built e9519654e875
@tianon Do you not forget to run your container? ls -la /clipboard/Gemfile.lock
is CMD command, which execute only on docker run
, not docker build
Oh bizarre -- something happens to the files when they get commit into a layer that breaks the second bundle install
. There's nothing different between the two invocations except Docker's layering. :confused:
For some reason I haven't figured out yet, when we run the second bundle install
in a new layer, bundle env
reports:
...
Set via BUNDLE_GEMFILE: "/ruby-jwt/Gemfile"
...
... even though no such environment variable is or ever was set, and we're in the /clipboard
directory, and it works fine if we run both bundle install
commands in the same layer without round-tripping through Docker.
Ok, I've managed to reproduce with @yosifkit's single-layer image as well by adding hash -r
in between the two bundle install
invocations. What's happening here is that the image-provided /usr/local/bin/bundle
script is getting shadowed by a new /usr/local/bundle/bin/bundle
script (via binstubs
) which is pointing at a specific Gemfile
instead.
@tianon as mentioned on previous comments, this isn't Docker fault by its own, but the combination of how this image configures Bundler (BUNDLE_BIN
is always set) and upstream changes on Bundler (see bundler/bundler#6154 and bundler/bundler#6162).
Applying the following simple patch appears to do the trick:
-ENV PATH $BUNDLE_BIN:$PATH
+
+# bundler itself is in /usr/local/bin, and due to https://github.com/docker-library/ruby/issues/188 (and thus https://github.com/bundler/bundler/issues/6162) we need to ensure that we don't get a bundler-binstub'd version of bundler in the PATH before our global bundler (hence the explicit "/usr/local/bin" prefix on this PATH)
+ENV PATH /usr/local/bin:$BUNDLE_BIN:$PATH
+
Sending build context to Docker daemon 2.048kB
Step 1/6 : FROM f43db8c0ae0b
---> f43db8c0ae0b
Step 2/6 : RUN git clone https://github.com/jwt/ruby-jwt.git
---> Running in 605212cce70f
Cloning into 'ruby-jwt'...
Removing intermediate container 605212cce70f
---> b96e0830368a
Step 3/6 : RUN cd ruby-jwt && bundle install
---> Running in 1fe833dc0c17
Fetching gem metadata from https://rubygems.org/........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Fetching rake 12.3.0
Installing rake 12.3.0
Using bundler 1.16.1
Fetching docile 1.1.5
Installing docile 1.1.5
Fetching json 2.1.0
Installing json 2.1.0 with native extensions
Fetching simplecov-html 0.10.2
Installing simplecov-html 0.10.2
Fetching simplecov 0.13.0
Installing simplecov 0.13.0
Fetching codacy-coverage 1.1.8
Installing codacy-coverage 1.1.8
Fetching codeclimate-test-reporter 1.0.8
Installing codeclimate-test-reporter 1.0.8
Fetching diff-lcs 1.3
Installing diff-lcs 1.3
Fetching ffi 1.9.23
Installing ffi 1.9.23 with native extensions
Using jwt 2.1.0 from source at `.`
Fetching rbnacl 5.0.0
Installing rbnacl 5.0.0
Fetching rspec-support 3.7.1
Installing rspec-support 3.7.1
Fetching rspec-core 3.7.1
Installing rspec-core 3.7.1
Fetching rspec-expectations 3.7.0
Installing rspec-expectations 3.7.0
Fetching rspec-mocks 3.7.0
Installing rspec-mocks 3.7.0
Fetching rspec 3.7.0
Installing rspec 3.7.0
Fetching simplecov-json 0.2
Installing simplecov-json 0.2
Bundle complete! 9 Gemfile dependencies, 18 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container 1fe833dc0c17
---> 9fd2e01b9635
Step 4/6 : RUN git clone https://github.com/janlelis/clipboard.git
---> Running in bcda7be744b8
Cloning into 'clipboard'...
Removing intermediate container bcda7be744b8
---> 9930c870c723
Step 5/6 : RUN cd clipboard && bundle install
---> Running in 4db88df3ebff
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Fetching rake 11.3.0
Installing rake 11.3.0
Using bundler 1.16.1
Using clipboard 1.1.1 from source at `.`
Using diff-lcs 1.3
Using rspec-support 3.7.1
Using rspec-core 3.7.1
Using rspec-expectations 3.7.0
Using rspec-mocks 3.7.0
Using rspec 3.7.0
Bundle complete! 4 Gemfile dependencies, 9 gems now installed.
Bundled gems are installed into `/usr/local/bundle`
Removing intermediate container 4db88df3ebff
---> 953376471d3b
Step 6/6 : RUN ls -la /clipboard/Gemfile.lock
---> Running in f7c7cfdd1c8f
-rw-r--r-- 1 root root 661 Feb 27 22:22 /clipboard/Gemfile.lock
Removing intermediate container f7c7cfdd1c8f
---> 1cd1366dee09
Successfully built 1cd1366dee09
To me that patch, or #191, are workarounds to the same root problem: this image shouldn't be setting those environment variables and should instead leave the default bundler environment untouched.
@deivid-rodriguez @lucasmazza @tianon I agree that we should be generalizing the image, rather than keep on adding configurations and patching.
Given the comment here: https://github.com/bundler/bundler/pull/6469#issuecomment-383235438. I think the best fix for the Docker image is this. It would keep it future-proof if the steps outlined there are followed.
-ENV PATH $BUNDLE_BIN:$PATH
+ENV PATH $BUNDLE_PATH/gems/bin:$PATH
I have successfully built the following Dockerfiles using this change that fail in the current ruby images:
FROM ruby:2.4-test
RUN set -ex;\
git clone https://github.com/jwt/ruby-jwt.git;\
cd ruby-jwt;\
bundle install
RUN set -ex;\
git clone https://github.com/janlelis/clipboard.git;\
cd clipboard;\
bundle install;\
ls -la /clipboard/Gemfile.lock
FROM ruby:2.4-test
RUN set -ex;\
git clone https://github.com/jwt/ruby-jwt.git;\
cd ruby-jwt;\
bundle install;\
cd ..;\
hash -r; \
git clone https://github.com/janlelis/clipboard.git;\
cd clipboard;\
bundle install;\
ls -la /clipboard/Gemfile.lock
If we do this, then we could probably revert or adjust the docs added in https://github.com/docker-library/docs/pull/1221.
Could those having this issue try setting this at the top of their Dockerfile to see if it works for them?
# this is the PATH before it is adjusted by the ruby Dockerfile
ENV PATH $BUNDLE_PATH/gems/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Hi, i notice some regression in update from
ruby:2.4.1
toruby:2.4.2
This dockerfileFor ruby 2.4.1 it correctly show presence of
/clipboard/Gemfile.lock
But if same Dockerimage createdFROM ruby:2.4.2
this causels: cannot access /clipboard/Gemfile.lock: No such file or directory
Probably somehow it related with update from
bundler 1.15.4
tobundler 1.16.0
but even if I manully rollback to bundler 1.15.4 inruby:2.4.2
error is the same. This is actual for any project, i chooseruby-jwt
andcliboard
as they pretty small.