GitLab service doesn't work on 19.09 if deployed by NixOps #68841

Closed mawis closed 5 years ago

Describe the bug After upgrading to the NixOS 19.09 channel the GitLab I deploy using NixOps doesn't work anymore. It seems to me, that a closure is missing.

The gitlab-sidekiq service logs the following and restarts continuously:

Sep 15 13:31:12 nahe.amessage.eu systemd[1]: Started gitlab-sidekiq.service.
Sep 15 13:31:14 nahe.amessage.eu sidekiq[4025]: WARNING: PID file creation will be removed in Sidekiq 6.0, see #4045. Please use a proper process supervisor to start and manage your services
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: Cannot load database configuration:
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: File "/nix/store/3rdc6hl3sx7g47pqims4m7cqdxc3vd8a-gitlab-12.1.6/share/gitlab/config/database.yml" is a symlink that does not point to a valid file
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/j1pxixg6vwv54hsh2dc194420lh1yx4d-ruby2.6.4-railties-5.2.3/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/paths.rb:214:in `block in existent'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/j1pxixg6vwv54hsh2dc194420lh1yx4d-ruby2.6.4-railties-5.2.3/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/paths.rb:210:in `select'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/j1pxixg6vwv54hsh2dc194420lh1yx4d-ruby2.6.4-railties-5.2.3/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/paths.rb:210:in `existent'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/j1pxixg6vwv54hsh2dc194420lh1yx4d-ruby2.6.4-railties-5.2.3/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/application/configuration.rb:166:in `database_configuration'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/dz7a5mac68g6g5szc2cnncdhylfpx1vd-ruby2.6.4-activerecord-5.2.3/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/railtie.rb:133:in `block (2 levels) in <class:Rail>
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hszh2lxqcd7k6krbbh3cvg90k5ric8ci-ruby2.6.4-activesupport-5.2.3/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:71:in `instance_eval'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hszh2lxqcd7k6krbbh3cvg90k5ric8ci-ruby2.6.4-activesupport-5.2.3/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:71:in `block in execute_hook'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hszh2lxqcd7k6krbbh3cvg90k5ric8ci-ruby2.6.4-activesupport-5.2.3/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:62:in `with_execution_contro>
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hszh2lxqcd7k6krbbh3cvg90k5ric8ci-ruby2.6.4-activesupport-5.2.3/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:67:in `execute_hook'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hszh2lxqcd7k6krbbh3cvg90k5ric8ci-ruby2.6.4-activesupport-5.2.3/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:43:in `block in on_load'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hszh2lxqcd7k6krbbh3cvg90k5ric8ci-ruby2.6.4-activesupport-5.2.3/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:42:in `each'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hszh2lxqcd7k6krbbh3cvg90k5ric8ci-ruby2.6.4-activesupport-5.2.3/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/lazy_load_hooks.rb:42:in `on_load'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/dz7a5mac68g6g5szc2cnncdhylfpx1vd-ruby2.6.4-activerecord-5.2.3/lib/ruby/gems/2.6.0/gems/activerecord-5.2.3/lib/active_record/railtie.rb:132:in `block in <class:Railtie>'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/j1pxixg6vwv54hsh2dc194420lh1yx4d-ruby2.6.4-railties-5.2.3/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `instance_exec'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/j1pxixg6vwv54hsh2dc194420lh1yx4d-ruby2.6.4-railties-5.2.3/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:32:in `run'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/j1pxixg6vwv54hsh2dc194420lh1yx4d-ruby2.6.4-railties-5.2.3/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:61:in `block in run_initializers'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hav12kpvy0c6n4vhc45wzxgqr7ci1nqv-ruby-2.6.4/lib/ruby/2.6.0/tsort.rb:228:in `block in tsort_each'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hav12kpvy0c6n4vhc45wzxgqr7ci1nqv-ruby-2.6.4/lib/ruby/2.6.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hav12kpvy0c6n4vhc45wzxgqr7ci1nqv-ruby-2.6.4/lib/ruby/2.6.0/tsort.rb:431:in `each_strongly_connected_component_from'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hav12kpvy0c6n4vhc45wzxgqr7ci1nqv-ruby-2.6.4/lib/ruby/2.6.0/tsort.rb:349:in `block in each_strongly_connected_component'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hav12kpvy0c6n4vhc45wzxgqr7ci1nqv-ruby-2.6.4/lib/ruby/2.6.0/tsort.rb:347:in `each'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hav12kpvy0c6n4vhc45wzxgqr7ci1nqv-ruby-2.6.4/lib/ruby/2.6.0/tsort.rb:347:in `call'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hav12kpvy0c6n4vhc45wzxgqr7ci1nqv-ruby-2.6.4/lib/ruby/2.6.0/tsort.rb:347:in `each_strongly_connected_component'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hav12kpvy0c6n4vhc45wzxgqr7ci1nqv-ruby-2.6.4/lib/ruby/2.6.0/tsort.rb:226:in `tsort_each'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/hav12kpvy0c6n4vhc45wzxgqr7ci1nqv-ruby-2.6.4/lib/ruby/2.6.0/tsort.rb:205:in `tsort_each'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/j1pxixg6vwv54hsh2dc194420lh1yx4d-ruby2.6.4-railties-5.2.3/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/initializable.rb:60:in `run_initializers'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/j1pxixg6vwv54hsh2dc194420lh1yx4d-ruby2.6.4-railties-5.2.3/lib/ruby/gems/2.6.0/gems/railties-5.2.3/lib/rails/application.rb:361:in `initialize!'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/3rdc6hl3sx7g47pqims4m7cqdxc3vd8a-gitlab-12.1.6/share/gitlab/config/environment.rb:6:in `<top (required)>'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/k9ic3qn24bcnjy63mva5xn2d4wrsjxy5-ruby2.6.4-sidekiq-5.2.7/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:288:in `require'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/k9ic3qn24bcnjy63mva5xn2d4wrsjxy5-ruby2.6.4-sidekiq-5.2.7/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:288:in `boot_system'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/k9ic3qn24bcnjy63mva5xn2d4wrsjxy5-ruby2.6.4-sidekiq-5.2.7/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/a273arinrgy5mcjdi4v4pw27d2sij78z-gitlab-env-12.1.6/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/a273arinrgy5mcjdi4v4pw27d2sij78z-gitlab-env-12.1.6/bin/sidekiq:18:in `load'
Sep 15 13:31:18 nahe.amessage.eu sidekiq[4025]: /nix/store/a273arinrgy5mcjdi4v4pw27d2sij78z-gitlab-env-12.1.6/bin/sidekiq:18:in `<main>'
Sep 15 13:31:18 nahe.amessage.eu systemd[1]: gitlab-sidekiq.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 13:31:18 nahe.amessage.eu systemd[1]: gitlab-sidekiq.service: Failed with result 'exit-code'.
Sep 15 13:31:18 nahe.amessage.eu systemd[1]: gitlab-sidekiq.service: Consumed 6.088s CPU time, no IP traffic.

/nix/store/3rdc6hl3sx7g47pqims4m7cqdxc3vd8a-gitlab-12.1.6/share/gitlab/config/database.yml links to /nix/store/c8cpnz5zyg0n5nbq3yfl3p9bagyqwak5-database.yml which doesn't exist on the server, that gets deployed with NixOps. On the computer that runs NixOps the closure gets built and is present.

Expected behavior The file /nix/store/c8cpnz5zyg0n5nbq3yfl3p9bagyqwak5-database.yml should get copied to the server by nixops deploy. The GitLab service should be working.

Additional context

This is the configuration I use for GitLab:

{
      services.gitlab = {
        enable = true;
        databasePasswordFile = "/etc/passwords/gitlabDatabase";
        initialRootPasswordFile = "/etc/passwords/gitlabInitialRoot";
        https = true;
        host = "git.amessage.eu";
        port = 443;
        user = "git";
        group = "git";
        databaseUsername = "git";
        smtp = {
          enable = true;
          port = 25;
          domain = "git.amessage.eu";
        };
        secrets = {
          dbFile = "/etc/passwords/gitlabSecretsDb";
          secretFile = "/etc/passwords/gitlabSecretsSecret";
          otpFile = "/etc/passwords/gitlabSecretsOtp";
          jwsFile = "/etc/passwords/gitlabSecretsJws";
        };
        extraConfig = {
          email_from = "XXXXXXXXX";
          email_display_name = "amessage Gitlab";
          email_reply_to = "XXXXXXXX";
          default_project_features = { builds = false; };
        };
      };
}

Metadata On the system running nixops:

system: "x86_64-linux"
host os: Linux 5.2.13, NixOS, 19.09beta181.64e38f246ae (Loris)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3
channels(root): "nixos-19.09beta181.64e38f246ae, unstable-19.09pre192418.e19054ab3cd"
channels(matthias): "home-manager-19.03"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

On the server being deployed by nixops:

system: "x86_64-linux"
host os: Linux 4.19.71, NixOS, 19.09beta181.64e38f246ae (Loris)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

attribute:
- gitlab
module:
- services.gitlab

It seem the closure that were missing is from the old configuration. I also found some other closures that were missing. But I think all of them shouldn't be there. It's the content of /var/gitlab/state/config that still points to these.

So at the moment it seems to me the problem is, that the gitlab service has problems migrating from a 19.03 configuration to a 19.09 configuration.

cc @talyz

I'm further investigating. It seem the pre-start script of gitlab.service is failing:

Sep 15 17:53:56 nahe.amessage.eu systemd[1]: Starting gitlab.service...
Sep 15 17:53:56 nahe.amessage.eu gvn7ks3r22v3hi101svwhlwx6vsivgbd-unit-script-gitlab-pre-start[30105]: mkdir -p /var/gitlab/state/home/.ssh: OK
Sep 15 17:53:56 nahe.amessage.eu gvn7ks3r22v3hi101svwhlwx6vsivgbd-unit-script-gitlab-pre-start[30105]: chmod 700 /var/gitlab/state/home/.ssh: OK
Sep 15 17:53:56 nahe.amessage.eu gvn7ks3r22v3hi101svwhlwx6vsivgbd-unit-script-gitlab-pre-start[30105]: /nix/store/gvn7ks3r22v3hi101svwhlwx6vsivgbd-unit-script-gitlab-pre-start: line 18: /etc/passwords/gitlabDatabase: Permission denied
Sep 15 17:53:56 nahe.amessage.eu gvn7ks3r22v3hi101svwhlwx6vsivgbd-unit-script-gitlab-pre-start[30105]: Database password was an empty string!
Sep 15 17:53:56 nahe.amessage.eu systemd[1]: gitlab.service: Control process exited, code=exited, status=1/FAILURE
Sep 15 17:53:56 nahe.amessage.eu systemd[1]: gitlab.service: Failed with result 'exit-code'.
Sep 15 17:53:56 nahe.amessage.eu systemd[1]: Failed to start gitlab.service.

So somehow the script cannot access the file that should contain my database password. Initially I had the file 0640 for git:git – but even after I made the file /etc/passwords/gitlabDatabase 0644 (i.e. world readable) I get the “Permission denied”.

Update: I had problems with the access rights of /etc/passwords, so the rights inside the folder had no effects.

Okay, it is working now again and I will close this issue. For reference I will document below what I had to do to get it working again.

I had to tweak some configuration settings. The following changes where quite clear because the existing configuration keys didn't exist anymore and I noticed fast, that I have to add a File to the end of the keys:

services.gitlab.databasePassword(File)
services.gitlab.initialRootPassword(File)
services.gitlab.secrets.*(File)

I'm deploying these files now with NixOps deployment.keys feature. I had to take care, that it's not sufficient that these files are accessible by the user running gitlab (git in my case), but this user also needs to have access to the folder I deploy the keys to. In case of NixOps I had to add the git user to the keys group: users.extraUsers.git.extraGroups = [ "keys" ];

Problems harder to find (for me):

The default setting of services.gitlab.databaseHost changed from 127.0.0.1 to empty. I think for postgres this means connection over a Unix socket. This didn't work for me as I only allow the user postgres to access the database over a Unix socket. So I needed to configure services.gitlab.databaseHost explicitly to 127.0.0.1 now.
My database user was gitlab while the system user was git which worked quite well in NixOS 19.03, but actually I didn't know that the database was accessed by gitlab. When deploying the configuration I got an error by the gitlab service expression, that services.gitlab.user and services.gitlab.databaseUsername must be the same. I did notice that the default for databaseUsername is gitlab so I changed it to git which I was expecting to be the database user I was using. Actually this user was also the owner of gitlab database on my PostgreSQL server, but git had a different password than gitlab had. So gitlab couldn't connect to the database. After changing the database password of the git user in PostgreSQL I finally git gitlab running again.

Thanks for reporting this!

The migration issues should hopefully be fixed by #68721.

The secret options are indeed changed and brakes upgrades by design. This, and some other changes, should probably be documented in the release notes, however.

The requirement that services.gitlab.user and services.gitlab.databaseUsername be identical is only valid if peer authentication is used and thus shouldn't apply to you. You get this error because you have services.gitlab.databaseCreateLocally enabled, which also provisions a database for you automatically. However, it feels like this shouldn't be possible, since there's no reason to deploy a local database that we won't connect to.

I'll address these issues in PRs I'm currently working on. :)

Indeed it could be that I had the message of having different user and databaseUsernameat the beginning and addressed that first. Changing the database back to 127.0.0.1 probably was at a later step and I just kept the changed username at that later stage.

Thanks for the feedback.

NixOS / nixpkgs

GitLab service doesn't work on 19.09 if deployed by NixOps #68841