docker-compose build never finishes

lupulin commented 3 years ago

Descriptive summary

HEAD

Rationale

I followed the directions to build the Docker image for Hyrax Development and it's taking hours to build.

Expected behavior

Build should take a few minutes at most

Actual behavior

Build is taking hours. Still hasn't finished.

Steps to reproduce the behavior

git clone https://github.com/samvera/hyrax.git
cd hyrax/
docker-compose build

no-reply commented 3 years ago

i'm not able to reproduce over here.

the behavior looks related to https://github.com/rubygems/bundler/issues/2125, but i can't think of any reason this would happen on your system but not mine.

jeremyf commented 3 years ago

From the tech call today, I'm working on duplicating this error.

lupulin commented 3 years ago

Mac OS 11.1 (20C69) Docker Desktop 3.0.1 (50773)

jeremyf commented 3 years ago

I'm testing on:

MacOS 10.15.7 (19H15) Docker Desktop 3.0.1 (50773)

At this point, my fan's happily spinning and I'm in the middle of gem installation. I'll see what comes of all of this.

jeremyf commented 3 years ago

Here's what I have thusfar with a relatively clean build.

fatal: not a git repository (or any of the parent directories): .git
bundler: failed to load command: rake (/usr/local/bin/rake)
Bundler::VersionConflict: Bundler could not find compatible versions for gem "faraday":
  In snapshot (Gemfile.lock):
    faraday (= 0.12.2)

  In Gemfile:
    hyrax was resolved to 3.0.0.pre.rc2, which depends on
      active-fedora (~> 13.1, >= 13.1.2) was resolved to 13.2.3, which depends on
        faraday (~> 0.12)

    hyrax was resolved to 3.0.0.pre.rc2, which depends on
      browse-everything (>= 0.16, < 2.0) was resolved to 1.1.2, which depends on
        googleauth (>= 0.6.6, < 1.0) was resolved to 0.14.0, which depends on
          faraday (>= 0.17.3, < 2.0)

    rsolr (>= 1.0, < 3) was resolved to 2.3.0, which depends on
      faraday (>= 0.9.0)

    hyrax was resolved to 3.0.0.pre.rc2, which depends on
      valkyrie (>= 2.1.1, < 3.0) was resolved to 2.1.1, which depends on
        faraday (< 1.0)

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
  /usr/local/lib/ruby/2.7.0/bundler/resolver.rb:59:in `rescue in start'
  /usr/local/lib/ruby/2.7.0/bundler/resolver.rb:45:in `start'
  /usr/local/lib/ruby/2.7.0/bundler/resolver.rb:22:in `resolve'
  /usr/local/lib/ruby/2.7.0/bundler/definition.rb:258:in `resolve'
  /usr/local/lib/ruby/2.7.0/bundler/definition.rb:170:in `specs'
  /usr/local/lib/ruby/2.7.0/bundler/definition.rb:237:in `specs_for'
  /usr/local/lib/ruby/2.7.0/bundler/definition.rb:226:in `requested_specs'
  /usr/local/lib/ruby/2.7.0/bundler/runtime.rb:101:in `block in definition_method'
  /usr/local/lib/ruby/2.7.0/bundler/runtime.rb:20:in `setup'
  /usr/local/lib/ruby/2.7.0/bundler.rb:149:in `setup'
  /usr/local/lib/ruby/2.7.0/bundler/setup.rb:20:in `block in <top (required)>'
  /usr/local/lib/ruby/2.7.0/bundler/ui/shell.rb:136:in `with_level'
  /usr/local/lib/ruby/2.7.0/bundler/ui/shell.rb:88:in `silence'
  /usr/local/lib/ruby/2.7.0/bundler/setup.rb:20:in `<top (required)>'

My next step is to see about upgrading the .dassie gems

no-reply commented 3 years ago

@jeremyf i'm curious about Gemfile.lock conflicts. there shouldn't be any lockfiles checked in.

i'd start by deleting them.

jeremyf commented 3 years ago

@no-reply I did just that.

I'm well along the way to a full install/build.

@lupulin A few questions:

What is the Ruby version you are using?
What is the branch and SHA of Hyrax git repository you're working from?
Do you have a Gemfile.lock file?
Do you have a .internal_test_app file?
Do you have a .dassie/Gemfile.lock file?

lupulin commented 3 years ago

What is the Ruby version you are using?

ruby 2.4.0p0 (2016-12-24 revision 57164) [x86_64-darwin18]

What is the branch and SHA of Hyrax git repository you're working from?

origin/master e2c7894f5c9cf941508ee848d70221b02657fb85

Do you have a Gemfile.lock file?

No

Do you have a .internal_test_app file?

No

Do you have a .dassie/Gemfile.lock file?

No

jeremyf commented 3 years ago

@lupulin I suspect that you need a later version of Ruby. My current round of testing this issue was using Ruby 2.7.1.

Our builds test 2.5.8, 2.6.6 and 2.7.1 - https://github.com/samvera/hyrax/blob/master/.circleci/config.yml#L128-L183

Our gemspec requires ruby >= 2.5 (https://github.com/samvera/hyrax/blob/master/hyrax.gemspec#L27); It's a bit something didn't fail for this.

I use https://github.com/rbenv/rbenv to manage my various ruby versions. I'd like to see if you encounter the problem on a 2.5.8 or greater version of Ruby.

lupulin commented 3 years ago

The ruby version of the Dockerfile is using 2.7.1.

Does it matter what ruby version I have running locally?

lupulin commented 3 years ago

Using rbenv to install 2.7.1 now.

no-reply commented 3 years ago

i think @lupulin is right that the ruby on their local system shouldn't be relevant.

lupulin commented 3 years ago

The errors on the screenshot I added to the initial issue says there are multiple sources in the Gemfile, but I only see one source but there are 2 Gemfiles, I'm wondering if there might be a Gemfile conflict somewhere...

.dassie/Gemfile and ./Gemfile

jeremyf commented 3 years ago

Ah, I see in the Dockerfile the RUBY_VERSION=2.7.1, so perhaps I sent you on a needless quest. (I'm only partially versed in Docker)

jeremyf commented 3 years ago

@lupulin I also have two Gemfiles (and expect both of them). However, when I was running the bundle install I saw a way to turn that situation from a warning to an error (it's long scrolled out of my term history). I'm trying to track that down.

jeremyf commented 3 years ago

@lupulin I also have two Gemfiles (and expect both of them). However, when I was running the bundle install I saw a way to turn that situation from a warning to an error (it's long scrolled out of my term history). I'm trying to track that down.

jeremyf commented 3 years ago

This is a bit of a moon shot:

I don't know if machine level bundle configs would be in play? For reference, I don't have the following global values on my machine:

Settings are listed in order of priority. The top value will be used.
github.https
Set for the current user (/Users/jfriesen/.bundle/config): "true"

gem.test
Set for the current user (/Users/jfriesen/.bundle/config): "rspec"

gem.mit
Set for the current user (/Users/jfriesen/.bundle/config): false

gem.coc
Set for the current user (/Users/jfriesen/.bundle/config): false

build.libxml-ruby
Set for the current user (/Users/jfriesen/.bundle/config): "--use-system-libraries --with-xml2-include=/usr/local/opt/libxml2/include/libxml2"

build.mysql2
Set for the current user (/Users/jfriesen/.bundle/config): "--with-mysql-config=/usr/local/opt/mysql@5.7/bin/mysql_config --with-ldflags=-L/usr/local/opt/openssl@1.1/lib --with-cppflags=-I/usr/local/opt/openssl@1.1/include --with-cflags=-I/usr/local/opt/openssl@1.1/include"

However, I think I may be troubleshooting at the machine level and not the docker level.

no-reply commented 3 years ago

@jeremyf yeah, i think this ought to be encapsulated at the container level.

@lupulin i'm wondering whether something has changed in the image tag upstream?

$ docker images ruby:2.7.1-alpine
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ruby                2.7.1-alpine        b94a71e90222        8 months ago        51.5MB

jeremyf commented 3 years ago

I have the following:

❯ docker images ruby:2.7.1-alpine
REPOSITORY   TAG            IMAGE ID       CREATED        SIZE
ruby         2.7.1-alpine   b46ea0bc5984   5 months ago   52.3MB

lupulin commented 3 years ago

@jeremyf I have the same Image ID b46ea0bc5984

$ docker images ruby:2.7.1-alpine
REPOSITORY   TAG            IMAGE ID       CREATED        SIZE
ruby         2.7.1-alpine   b46ea0bc5984   5 months ago   52.3MB

no-reply commented 3 years ago

while we're here, i'm kicking at bumping us to ruby 2.7.2 https://github.com/samvera/hyrax/tree/rb2.7.2

jeremyf commented 3 years ago

@lupulin I would hope that it's not a Mac OSX version.

lupulin commented 3 years ago

@jeremyf lol. it very well could be... i've had weirder things fail after a mac upgrade.

Although, I've tried on Debian on a VM and it still didn't work for me.

no-reply commented 3 years ago

@jeremyf i was going to ask if you are also using OSX. i know docker on mac is a bit of a wild card, and there might be issues there.

@lupulin i want to just double check that you're doing exactly the three-step process you list in the main issue text:

git clone https://github.com/samvera/hyrax.git
cd hyrax/
docker-compose build

there's nothing else that could be muddying the waters. (sorry that this is a remedial question, but i guess this is where i'm at...)

jeremyf commented 3 years ago

Now might be the moment when I rant about software…

It all sucks.

lupulin commented 3 years ago

@no-reply I'll try again.

Can you verify that you're using a brand new git pull and verify that it works after deleting your Docker images?

jeremyf commented 3 years ago

@no-reply I'm on OS X; I have Docker Desktop freshly installed.

One interesting thing, yesterday Docker Desktop prompted me to upgrade. I did that, and Docker Desktop refused to complete it's start sequence. I tried to clean things up, but it wasn't having any of that. So I removed Docker Desktop (via App Cleaner) and reinstalled.

Needless to say, my laptops been a hot screaming (cpu-)fan boy much of today.

lupulin commented 3 years ago

@no-reply I tried the steps again (deleted the original repo) and I'm getting the same issue.

jeremyf commented 3 years ago

@lupulin I have 7ab558853e898f10aaeb9571463465dceb407029 as the HEAD of my master branch.

I also made it all the way through to a Rails app running on http://localhost:3000

no-reply commented 3 years ago

Can you verify that you're using a brand new git pull and verify that it works after deleting your Docker images?

yep. i was able to get this going with a brand new Arch Linux install on new hardware yesterday---i'm working on transferring my dev env to a new laptop at the moment.

on this machine, i'm doing docker container prune; docker image prune and trying with a fresh checkout now. i'll report back when that's done, but i'm not expecting different outcomes at this point. i'm at a bit of a loss.

really appreciate you reporting this and working with us on getting diagnosed.

no-reply commented 3 years ago

while i'm dumping system info:


tjohnson@hopper ~> docker-compose --version
docker-compose version 1.26.2, build unknown
tjohnson@hopper ~> docker version
Client:
 Version:           19.03.12-ce
 API version:       1.40
 Go version:        go1.14.5
 Git commit:        48a66213fe
 Built:             Sat Jul 18 01:33:21 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.12-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.14.5
  Git commit:       48a66213fe
  Built:            Sat Jul 18 01:32:59 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.4.m
  GitCommit:        d76c121f76a5fc8a462dc64594aea72fe18e1178.m
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683```

lupulin commented 3 years ago

$ docker-compose --version
docker-compose version 1.27.4, build 40524192
🍺 /Users/chris/Source/work/hyrax
$ docker version
Client: Docker Engine - Community
 Cloud integration: 1.0.4
 Version:           20.10.0
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        7287ab3
 Built:             Tue Dec  8 18:55:43 2020
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       eeddea2
  Built:            Tue Dec  8 18:58:04 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

jeremyf commented 3 years ago

❯ docker-compose --version
docker-compose version 1.27.4, build 40524192

❯ docker version
Client: Docker Engine - Community
 Cloud integration: 1.0.4
 Version:           20.10.0
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        7287ab3
 Built:             Tue Dec  8 18:55:43 2020
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.0
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       eeddea2
  Built:            Tue Dec  8 18:58:04 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

jeremyf commented 3 years ago

@lupulin Well…that's disheartening. We have the same versions

❯ docker-compose version
docker-compose version 1.27.4, build 40524192
docker-py version: 4.3.1
CPython version: 3.7.7
OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020

lupulin commented 3 years ago

I'm not sure what's going on with my environment. I tried uninstalling / reinstalling Docker, didn't help... I need to take a break. I assume I'm doing something wrong or my environment is off somehow.

jeremyf commented 3 years ago

@lupulin Thanks for working through this with us

no-reply commented 3 years ago

i was able to reproduce this today! on an existing Hyrax install i got the behavior descried after not building for a few weeks.

i was able to work around it by doing bundle on my local machine, in the hyrax repository root. after that finished docker-compose build worked okay.

i'm reopening this, and hopefully we can move closer to a real solution.

eporter23 commented 1 year ago

Closing this as part of older ticket reviews but please reopen if needed. There have been multiple updates to the Hyrax in a container troubleshooting since the last comments on this issue.

samvera / hyrax