Runners try to register before Gitlab installed and/or ready

bschonec commented 2 years ago

Affected Puppet, Ruby, OS and module versions/distributions

Puppet:

puppet-agent-7.15.0-1.el8.x86_64 puppetserver-7.6.1-1.el7.noarch

Ruby:
Distribution: Puppet server: Red Hat Enterprise Linux Server release 7.9 (Maipo) Puppet agent: Red Hat Enterprise Linux release 8.1 (Ootpa)
Module version: 4.3.0

How to reproduce (e.g Puppet code you use)

Puppet manifest:

  # Resource ordering  THIS DOESN'T ORDER PROPERLY
  Class['gitlab'] -> Class['gitlab_ci_runner'] 
  include ::gitlab

  # Install Gitlab CI Runners
  include ::gitlab_ci_runner

Hiera data:

gitlab_ci_runner::concurrent: 4
gitlab_ci_runner::check_interval: 4
gitlab_ci_runner::metrics_server: "localhost:8888"
gitlab_ci_runner::manage_docker: true
gitlab_ci_runner::config_path: "/etc/gitlab-runner/config.toml"
gitlab_ci_runner::runners:
  ubi8:
    ensure: present
    url: "http://10.10.10.10/"
    registration-token: "token"
    tag-list: "aws,docker,example-tag"
    executor: "docker"
    docker:
      image: "ubi/ubi8:minimal"
  alpine:
    ensure: present
    url: "http://10.10.10.10/"
    registration-token: "token"
    tag-list: "alpine"
    executor: "docker"
    docker:
      image: "alpine:latest"

gitlab_ci_runner::runner_defaults:
  url: "https://localhost/ci"
  registration-token: "token"
  executor: "docker"
  docker:
    image: "ubi/ubi7:minimal"

What are you seeing

When running the puppet manifest, the gitlab_ci_runner https://github.com/voxpupuli/puppet-gitlab_ci_runner/blob/master/manifests/init.pp#L139 tries to connect to the local machine before gitlab is installed from the include ::gitlab code.

What behaviour did you expect instead

I expected Gitlab class module to run and install Gitlab CE and then the gitlab_ci_runner class module register the runners.

Output log

Info: Using environment '23_create_gitlab_ce_role_profile_for_installing_and_managing_gitlabce'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Failed to apply catalog: Failed to open TCP connection to 10.100.100.10:80 (Connection refused - connect(2) for "10.100.100.119" port 80)

Any additional information you'd like to impart

Of course, since Gitlab isn't installed there's no way to configure the runners.

I've been out of the Puppet world for about four years so I'm rusty. I'm not familiar with Deferred Functions.

Is the resource ordering above not correct? Is there a way in the puppet module to ensure that the runners are registered later/last?

tuxmea commented 2 years ago

Have you tried using contain instead of include?

  # Resource ordering  THIS DOESN'T ORDER PROPERLY
  Class['gitlab'] -> Class['gitlab_ci_runner'] 
  contain ::gitlab

  # Install Gitlab CI Runners
  contain ::gitlab_ci_runner

bschonec commented 2 years ago

I've tried your suggestion but I'm getting the same results. I did notice that the puppet run is failing at https://github.com/voxpupuli/puppet-gitlab_ci_runner/blob/master/lib/puppet/functions/gitlab_ci_runner/register_to_file.rb#L44 before the class module even has a chance to do anything or that resource ordering is enforced.

alexjfisher commented 2 years ago

@tuxmea The registering takes place in a Deferred function that gets run before any of the rest of the catalog is actually applied.

If you're installing a runner on the same host as gitlab is being installed, then you might want to wrap the gitlab runner part of your puppet code in a fact that checks if gitlab itself has been installed yet. Doesn't look like https://github.com/voxpupuli/puppet-gitlab contains any facts, but creating a simple gitlab_installed fact should be pretty trivial.

Then your puppet code might look a bit like...

include gitlab

if $facts['gitlab_installed'] {
  include gitlab_ci_runner
} else {
  notify { 'Rerun puppet to install gitlab runner':}
}

Your custom fact (which you could put in your profile module) may be as simple as something like ...

# profile/lib/facter/gitlab_installed.rb
Facter.add(:gitlab_installed) do
  setcode do
    File.exist?('/path/to/some/file/that/exists/after/gitlab/is/installed')
  end
end

Hope that helps.

alexjfisher commented 2 years ago

@tuxmea Also see https://tickets.puppetlabs.com/browse/PUP-9323

alexjfisher commented 2 years ago

Just spotted this 'draft' PR. https://github.com/puppetlabs/puppet/pull/8902 :)

joshcooper commented 2 years ago

:wave: Hope to have the deferred PR more ready to go next week. It'll fix two important limitations with Deferred functions.

Currently if a deferred function fails, then the entire run fails. With the PR, only the resource will fail (and other resources that depend on it, direectly or transitively).
Currently it's not possible to install a prerequisite (like a gem) for a deferred function and call the deferred function in the same run. However, due to the above limitation, you can't install the gem and run the agent again. In other words, it never converges.

jkroepke commented 2 years ago

Hi, this problem still exists in puppet 7.19...

# /opt/puppetlabs/bin/puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Failed to apply catalog: Failed to open TCP connection to gitlab:80 (Connection refused - connect(2) for "gitlab" port 80)

here is a trace:

Error: Failed to apply catalog: Failed to open TCP connection to gitlab:80 (Connection refused - connect(2) for "gitlab" port 80)
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:960:in `initialize'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:960:in `open'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:960:in `block in connect'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:105:in `timeout'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:958:in `connect'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:943:in `do_start'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:932:in `start'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/net/http.rb:1483:in `request'
/opt/puppetlabs/puppet/cache/lib/puppet_x/gitlab/runner.rb:47:in `request'
/opt/puppetlabs/puppet/cache/lib/puppet_x/gitlab/runner.rb:18:in `post'
/opt/puppetlabs/puppet/cache/lib/puppet_x/gitlab/runner.rb:58:in `register'
/opt/puppetlabs/puppet/cache/lib/puppet/functions/gitlab_ci_runner/register_to_file.rb:44:in `register_to_file'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:60:in `invoke'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:47:in `block in dispatch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:46:in `catch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:46:in `dispatch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:46:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:45:in `catch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:45:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:283:in `block in external_call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:282:in `external_call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/scope.rb:1102:in `call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:151:in `resolve_future'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:102:in `resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `block in resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `each_pair'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:104:in `block in resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:104:in `map'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:104:in `resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `block in resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `each_pair'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:107:in `resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:166:in `block in map_arguments'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:166:in `map'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:166:in `map_arguments'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:141:in `resolve_future'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:102:in `resolve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:84:in `block (2 levels) in resolve_futures'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:83:in `each_pair'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:83:in `block in resolve_futures'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:81:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:81:in `resolve_futures'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/deferred_resolver.rb:44:in `resolve_and_replace'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:115:in `block in convert_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:567:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:566:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:112:in `convert_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:481:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:339:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:83:in `block (6 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:82:in `block (5 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/timeout.rb:105:in `timeout'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:81:in `block (4 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent/locker.rb:21:in `lock'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:71:in `block (3 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:162:in `with_client'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:67:in `block (2 levels) in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:127:in `run_in_fork'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:66:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:172:in `controlled_run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/agent.rb:47:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:433:in `onetime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:393:in `block in run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/agent.rb:390:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:421:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:756:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:421:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
Debug: Closing connection for https://puppet:8140

mojibake-umd commented 10 months ago

I would like to comment that I just tested setting [agent] preprocess_deferred = false in puppet.conf or via hiera. puppet::agent_additional_settings: # Disable deferred functions # gitlab_ci_runner will try to register before # anything is even installed and configured. preprocess_deferred: false

Which seems to get around the issue for me of launching an EC2 runner from zero.

I tested against 4.4.0. puppet 7.24.0

EDIT:

Actually that only gets you a little further along in puppet setup. End up running into https://github.com/puppetlabs/puppet/issues/9174 and still failing to get gitlab runner registered.

Error: /Stage[main]/Gitlab_ci_runner::Config/Concat[/etc/gitlab-runner/config.toml]/Concat_file[/etc/gitlab-runner/config.toml]: Failed to generate additional resources using 'eval_generate': undefined method `<<' for #<Puppet::Pops::Evaluator::DeferredValue:0x00000000087d1df8>

voxpupuli / puppet-gitlab_ci_runner