samvera / hyku

Hyku: A multi-tenant Hyrax application built on the latest and greatest Samvera community components. Brought to you by the Hydra-in-a-Box project partners and IMLS; maintained by the Hyku Interest Group.
https://samvera.atlassian.net/wiki/spaces/hyku/overview
Other
96 stars 49 forks source link

Docker - Launching stack - service "initialize_app" fails #1803

Open keebeegee opened 2 years ago

keebeegee commented 2 years ago

Problem: service "initialize_app" didn't completed successfully and therefore prevent other services from starting

Expected behavior: service "initialize_app" completes successfully so that the following services will be started

Environment:

Output:

kbg@cx30-580953:~/hyku$ docker-compose up -d web
WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string.
[+] Running 19/19
 ⠿ Network hyku_internal            Created                                                                                                                                                                                                                                                                            0.1s
 ⠿ Volume "hyku_redis"              Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_solr"               Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_zk"                 Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_uploads"            Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_assets"             Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_cache"              Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_fcrepo"             Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Volume "hyku_db"                 Created                                                                                                                                                                                                                                                                            0.0s
 ⠿ Container hyku-redis-1           Started                                                                                                                                                                                                                                                                            5.5s
 ⠿ Container hyku-zoo-1             Healthy                                                                                                                                                                                                                                                                           17.9s
 ⠿ Container hyku-check_volumes-1   Started                                                                                                                                                                                                                                                                            5.5s
 ⠿ Container hyku-db-1              Started                                                                                                                                                                                                                                                                            5.4s
 ⠿ Container hyku-chrome-1          Started                                                                                                                                                                                                                                                                            4.8s
 ⠿ Container hyku-fcrepo-1          Started                                                                                                                                                                                                                                                                            4.7s
 ⠿ Container hyku-solr-1            Healthy                                                                                                                                                                                                                                                                          100.7s
 ⠿ Container hyku-initialize_app-1  Exited                                                                                                                                                                                                                                                                           161.3s
 ⠿ Container hyku-worker-1          Created                                                                                                                                                                                                                                                                            0.2s
 ⠿ Container hyku-web-1             Created  

service "initialize_app" didn't completed successfully: exit 1

See below excerpts from log (initialize_app), see full log here: log.txt

...

hyku-initialize_app-1  | ERROR:  relation "schema_migrations" does not exist
hyku-initialize_app-1  | LINE 1: SELECT version FROM schema_migrations ORDER BY schema_migrat...

....

Migrating to AddThemesToSites (20210121233027)
   (0.3ms)  BEGIN
== 20210121233027 AddThemesToSites: migrating =================================
-- add_column(:sites, :home_theme, :string)
   (0.5ms)  ALTER TABLE "sites" ADD "home_theme" character varying
   -> 0.0009s
rails aborted!
Errno::EACCES: Permission denied @ rb_sysopen - /app/samvera/hyrax-webapp/db/schema.rb
/usr/local/bundle/gems/activerecord-5.2.8/lib/active_record/railties/databases.rake:250:in `initialize'
/usr/local/bundle/gems/activerecord-5.2.8/lib/active_record/railties/databases.rake:250:in `open'
/usr/local/bundle/gems/activerecord-5.2.8/lib/active_record/railties/databases.rake:250:in `block (3 levels) in <top (required)>'
/usr/local/bundle/gems/activerecord-5.2.8/lib/active_record/railties/databases.rake:68:in `block (2 levels) in <top (required)>'
/usr/local/bundle/gems/activerecord-5.2.8/lib/active_record/railties/databases.rake:61:in `block (2 levels) in <top (required)>'
/usr/local/bundle/gems/railties-5.2.8/lib/rails/commands/rake/rake_command.rb:23:in `block in perform'
/usr/local/bundle/gems/railties-5.2.8/lib/rails/commands/rake/rake_command.rb:20:in `perform'
/usr/local/bundle/gems/railties-5.2.8/lib/rails/command.rb:48:in `invoke'
/usr/local/bundle/gems/railties-5.2.8/lib/rails/commands.rb:18:in `<top (required)>'
bin/rails:4:in `require'
bin/rails:4:in `<main>'
Tasks: TOP => db:schema:dump
(See full trace by running task with --trace)
-- add_column(:sites, :show_theme, :string)
   (0.6ms)  ALTER TABLE "sites" ADD "show_theme" character varying
   -> 0.0010s
-- add_column(:sites, :search_theme, :string)
   (0.6ms)  ALTER TABLE "sites" ADD "search_theme" character varying
   -> 0.0008s
== 20210121233027 AddThemesToSites: migrated (0.0030s) ========================
...

Observations:

a) The error

        rails aborted!
        Errno::EACCES: Permission denied @ rb_sysopen - /app/samvera/hyrax-webapp/db/schema.rb

seems to be triggered in all cases during execution of the following migration step:

hyku/db/migrate/20210121233027_add_themes_to_sites.rb

b) Migration steps located before and after "20210121233027_add_themes_to_sites.rb" are executed without triggering any errors.

c) Everything seems to work fine in Ubuntu 20.04 - WSL2. Containers start without error:

....  
=> => naming to ghcr.io/samvera/hyku:latest                                                                       0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
[+] Running 19/19
 ⠿ Network hyku_internal            Created                                                                        0.8s
 ⠿ Volume "hyku_cache"              Created                                                                        0.0s
 ⠿ Volume "hyku_fcrepo"             Created                                                                        0.0s
 ⠿ Volume "hyku_solr"               Created                                                                        0.0s
 ⠿ Volume "hyku_zk"                 Created                                                                        0.0s
 ⠿ Volume "hyku_redis"              Created                                                                        0.0s
 ⠿ Volume "hyku_db"                 Created                                                                        0.0s
 ⠿ Volume "hyku_uploads"            Created                                                                        0.0s
 ⠿ Volume "hyku_assets"             Created                                                                        0.0s
 ⠿ Container hyku-fcrepo-1          Started                                                                        7.9s
 ⠿ Container hyku-db-1              Started                                                                        5.9s
 ⠿ Container hyku-check_volumes-1   Started                                                                        8.2s
 ⠿ Container hyku-redis-1           Started                                                                        8.1s
 ⠿ Container hyku-chrome-1          Started                                                                        7.7s
 ⠿ Container hyku-zoo-1             Healthy                                                                       20.6s
 ⠿ Container hyku-solr-1            Healthy                                                                       84.8s
 ⠿ Container hyku-initialize_app-1  Exited                                                                       209.4s
 ⠿ Container hyku-worker-1          Started                                                                      210.1s
 ⠿ Container hyku-web-1             Started
xx@DESKTOP-269UBN3:/mnt/d/xxx/xxx/hyku$
keebeegee commented 2 years ago

Permission issue most likely caused by ignoring "newgrp docker"

orangewolf commented 2 years ago

thank you for your report =-)

I think there are fundamentally two issues here. The first one is that we need add a migration to the schema.rb file and commit that in main. When you start up fresh it should load the schema and not need a migration right off the bat, but if one is not checked in then db/schema.rb file will be modified.

Second, and I think this is the bigger issue in regards to running in your original configuration is that inside the container, the db/schema.rb file can not be written too. This means that the docker-compose file mount is not getting the correct permissions. In both MacOS and Windows, bind mounts are handled by a fairly complex mount/sync process. My understanding is that in native Linux this process is simpler. I'm wondering, if you were to run docker-compose run initialize_app sh and look inside the container, are all the file permissions incorrect? Does your code checkout outside the container have the same user / permissions as the files inside it?

I do not have a lot of experience running docker-compose with bind mounts on Linux hosts (we mostly use Linux hosts for deployments and bind mounts are not common there). The first issue is one I'll take care of, but the if you can provide more information about the second it sounds like we at least need to add an optional step to the README.

keebeegee commented 2 years ago

Dear @orangewolf thanks for your input. I am re-opening this issue again as a proper start was only achieved once out of ~30 times. All subsequent tries after the single successful one failed as described above. Removing all images and re-creating them would not change anything.

Just to be clear: The err as described above occurred on a Ubuntu 20.04 LTS. I later tried to run the stack on a WIN WSL 2 (Ubuntu 20.04 LTS) and all containers started properly.

Will look into your advise - thanks again for looking into this.

adamhgriffith-uofu commented 1 year ago

Hi @keebeegee and @orangewolf ,

On Linux hosts, the file permissions set on content in the Docker volume are identical from the perspective of the host as well as the container. Since the Dockerfile is rootless and runs as the app user, it doesn't have the permissions to read the mounted volume:

x-app: &app
  ...
  volumes:
   ...
    - .:/app/samvera/hyrax-webapp     <---------------------------------------------

Our group is also exploring this stack on our Linux workstations and found that commenting out the line indicated above resulted in a working environment.

orangewolf commented 1 year ago

@adamhgriffith-uofu yes, if you remove the bind mount it works ok... but you don't get to edit code outside the container and have it work inside the container. That's ok for a production deployment but makes development very awkward. The good news is that we have a new solution for this using the user id that seems to be successful more uniformly. I'll try to get a PR up about it early next week.

ryan-lang commented 5 months ago

@orangewolf Have there been any updates on this issue?