search

Esri / arcgis-cookbook

Chef cookbooks for ArcGIS
Apache License 2.0
300 stars 116 forks source link

Docker file not found in in Chef ArcGIS 4.0.0 Notebooks 10.9.1 #393

Closed df-sloughton closed 3 weeks ago

df-sloughton commented 4 months ago

Has something potentially happened recently for docker not being made available via chef arcgis 4.0.0 This has deployed successfully for months and now it wont.

OS: Ubuntu Linux 22.02 ArcGIS Enterprise 10.9.1 Chef ArcGIS Cookbooks 4.0.0

[2024-08-02T06:23:45+00:00] INFO: Loading cookbooks [arcgis-enterprise@4.0.0, arcgis-notebooks@4.0.0, arcgis-repository@4.0.0, hostsfile@3.0.1, limits@1.0.0, windows@5.3.1, windows_firewall@5.0.1, ms_dotnet@4.2.1, nfs@2.6.4, java_properties@0.1.3, docker@4.12.0, iptables@7.1.0, s3_file@2.8.5, line@2.9.3]
Synchronizing cookbooks:
  - arcgis-enterprise (4.0.0)
  - arcgis-notebooks (4.0.0)
  - arcgis-repository (4.0.0)
  - hostsfile (3.0.1)
  - limits (1.0.0)
  - windows_firewall (5.0.1)
  - windows (5.3.1)
  - ms_dotnet (4.2.1)
  - nfs (2.6.4)
  - java_properties (0.1.3)
  - docker (4.12.0)
  - iptables (7.1.0)
  - s3_file (2.8.5)
  - line (2.9.3)
Installing cookbook gem dependencies:
[2024-08-02T06:23:46+00:00] INFO: Fetching gem metadata from https://rubygems.org/........
Resolving dependencies...
Using bundler 2.3.7
Using multi_json 1.15.0
Using excon 0.111.0
Using docker-api 1.34.2
Bundle complete! 1 Gemfile dependency, 4 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

Compiling cookbooks...
[2024-08-02T06:23:46+00:00] DEBUG: Cookbooks to compile: [:line, :nfs, :s3_file, :"arcgis-repository", :hostsfile, :java_properties, :limits, :windows, :ms_dotnet, :windows_firewall, :"arcgis-enterprise", :docker, :iptables, :"arcgis-notebooks"]

================================================================================
Recipe Compile Error in /root/.cinc/local-mode-cache/cache/cookbooks/docker/libraries/docker_base.rb
================================================================================

FrozenError
-----------
can't modify frozen Array: [:allow_unstubbed_requests, :body, :chunk_size, :debug_request, :debug_response, :dns_timeouts, :headers, :instrumentor, :logger, :method, :middlewares, :password, :path, :persistent, :pipeline, :query,:read_timeout, :request_block, :resolv_resolver, :response_block, :stubs, :timeout, :user, :versions, :write_timeout]

Cookbook Trace: (most recent call first)
----------------------------------------
  /root/.cinc/local-mode-cache/cache/cookbooks/docker/libraries/docker_base.rb:3:in `<class:DockerBase>'
  /root/.cinc/local-mode-cache/cache/cookbooks/docker/libraries/docker_base.rb:2:in `<module:DockerCookbook>'
  /root/.cinc/local-mode-cache/cache/cookbooks/docker/libraries/docker_base.rb:1:in `<top (required)>'

Relevant File Content:
----------------------
/root/.cinc/local-mode-cache/cache/cookbooks/docker/libraries/docker_base.rb:

  1:  module DockerCookbook
  2:    class DockerBase < Chef::Resource
  3>>     require 'docker'
  4:      require 'shellwords'
  5:
  6:      ################
  7:      # Helper methods
  8:      ################
  9:
 10:      def connection
 11:        @connection ||= begin
 12:                          opts = {}

Additional information:
-----------------------
      Ruby objects are often frozen to prevent further modifications
      when they would negatively impact the process (e.g. values inside
      Ruby's ENV class) or to prevent polluting other objects when default
      values are passed by reference to many instances of an object (e.g.
      the empty Array as a Chef resource default, passed by reference
      to every instance of the resource).

      Chef uses Object#freeze to ensure the default values of properties
      inside Chef resources are not modified, so that when a new instance
      of a Chef resource is created, and Object#dup copies values by
      reference, the new resource is not receiving a default value that
      has been by a previous instance of that resource.

      Instead of modifying an object that contains a default value for all
      instances of a Chef resource, create a new object and assign it to
      the resource's parameter, e.g.:

      fruit_basket = resource(:fruit_basket, 'default')

      # BAD: modifies 'contents' object for all new fruit_basket instances
      fruit_basket.contents << 'apple'

      # GOOD: allocates new array only owned by this fruit_basket instance
      fruit_basket.contents %w(apple)

System Info:
------------
chef_version=18.2.7
platform=ubuntu
platform_version=22.04
ruby=ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
program_name=/usr/bin/cinc-client
executable=/opt/cinc/bin/cinc-client

Running handlers:
[2024-08-02T06:23:46+00:00] ERROR: Running exception handlers
Running handlers complete
[2024-08-02T06:23:46+00:00] ERROR: Exception handlers complete
Infra Phase failed. 0 resources updated in 03 seconds
[2024-08-02T06:23:46+00:00] DEBUG: Chef::Compliance::Runner#enabled?: Cinc Auditor profiles? false
[2024-08-02T06:23:46+00:00] DEBUG: Chef::Compliance::Runner#enabled?: audit cookbook?
[2024-08-02T06:23:46+00:00] DEBUG: Chef::Compliance::Runner#enabled?: compliance phase attr? false
[2024-08-02T06:23:46+00:00] FATAL: Stacktrace dumped to /root/.cinc/local-mode-cache/cache/cinc-stacktrace.out
[2024-08-02T06:23:46+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-08-02T06:23:46+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2024-08-02T06:23:46+00:00] FATAL: ---------------------------------------------------------------------------------------
[2024-08-02T06:23:46+00:00] DEBUG: FrozenError: can't modify frozen Array: [:allow_unstubbed_requests, :body, :chunk_size, :debug_request, :debug_response, :dns_timeouts, :headers, :instrumentor, :logger, :method, :middlewares, :password, :path, :persistent, :pipeline, :query, :read_timeout, :request_block, :resolv_resolver, :response_block, :stubs, :timeout, :user, :versions, :write_timeout]
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/docker-api-1.34.2/lib/excon/middlewares/hijack.rb:2:in `<module:Excon>'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/docker-api-1.34.2/lib/excon/middlewares/hijack.rb:1:in `<top (required)>'
<internal:/opt/cinc/embedded/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
<internal:/opt/cinc/embedded/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/docker-api-1.34.2/lib/docker.rb:14:in `<top (required)>'
<internal:/opt/cinc/embedded/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:160:in `require'
<internal:/opt/cinc/embedded/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
<internal:/opt/cinc/embedded/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
/root/.cinc/local-mode-cache/cache/cookbooks/docker/libraries/docker_base.rb:3:in `<class:DockerBase>'
/root/.cinc/local-mode-cache/cache/cookbooks/docker/libraries/docker_base.rb:2:in `<module:DockerCookbook>'
/root/.cinc/local-mode-cache/cache/cookbooks/docker/libraries/docker_base.rb:1:in `<top (required)>'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:290:in `require'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:290:in `block in load_libraries_from_cookbook'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:431:in `block (2 levels) in each_file_in_cookbook_by_segment'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:428:in `each'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:428:in `block in each_file_in_cookbook_by_segment'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:427:in `each'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:427:in `each_file_in_cookbook_by_segment'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:287:in `load_libraries_from_cookbook'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:135:in `block in compile_libraries'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:132:in `each'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:132:in `compile_libraries'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:103:in `compile'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context.rb:259:in `load'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/policy_builder/expand_node_object.rb:103:in `setup_run_context'
/opt/cinc/embedded/lib/ruby/3.1.0/forwardable.rb:238:in `setup_run_context'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/client.rb:508:in `setup_run_context'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/client.rb:294:in `run'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/application.rb:264:in `run_chef_client'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/application/base.rb:352:in `run_application'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/application.rb:67:in `run'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-18.2.7/bin/cinc-client:25:in `<top (required)>'
/usr/bin/cinc-client:180:in `load'
/usr/bin/cinc-client:180:in `<main>'

>>>> Caused by LoadError: cannot load such file -- docker
<internal:/opt/cinc/embedded/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
<internal:/opt/cinc/embedded/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
/root/.cinc/local-mode-cache/cache/cookbooks/docker/libraries/docker_base.rb:3:in `<class:DockerBase>'
/root/.cinc/local-mode-cache/cache/cookbooks/docker/libraries/docker_base.rb:2:in `<module:DockerCookbook>'
/root/.cinc/local-mode-cache/cache/cookbooks/docker/libraries/docker_base.rb:1:in `<top (required)>'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:290:in `require'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:290:in `block in load_libraries_from_cookbook'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:431:in `block (2 levels) in each_file_in_cookbook_by_segment'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:428:in `each'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:428:in `block in each_file_in_cookbook_by_segment'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:427:in `each'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:427:in `each_file_in_cookbook_by_segment'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:287:in `load_libraries_from_cookbook'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:135:in `block in compile_libraries'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:132:in `each'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:132:in `compile_libraries'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context/cookbook_compiler.rb:103:in `compile'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/run_context.rb:259:in `load'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/policy_builder/expand_node_object.rb:103:in `setup_run_context'
/opt/cinc/embedded/lib/ruby/3.1.0/forwardable.rb:238:in `setup_run_context'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/client.rb:508:in `setup_run_context'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/client.rb:294:in `run'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/application.rb:305:in `run_with_graceful_exit_option'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/application.rb:281:in `block in run_chef_client'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/local_mode.rb:42:in `with_server_connectivity'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/application.rb:264:in `run_chef_client'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/application/base.rb:352:in `run_application'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-18.2.7/lib/chef/application.rb:67:in `run'
/opt/cinc/embedded/lib/ruby/gems/3.1.0/gems/chef-bin-18.2.7/bin/cinc-client:25:in `<top (required)>'
/usr/bin/cinc-client:180:in `load'
/usr/bin/cinc-client:180:in `<main>'
[2024-08-02T06:23:46+00:00] FATAL: FrozenError: can't modify frozen Array: [:allow_unstubbed_requests, :body, :chunk_size, :debug_request, :debug_response, :dns_timeouts, :headers, :instrumentor, :logger, :method, :middlewares, :password, :path, :persistent, :pipeline, :query, :read_timeout, :request_block, :resolv_resolver, :response_block, :stubs, :timeout, :user, :versions, :write_timeout]
ERROR: cinc-client failed
ERROR: Failed to initialise role: Server notebook

Docker-api is there in gen list:

 root@x-y-z-w:~# /opt/cinc/bin/gem list | grep docker
docker-api (1.34.2)
cameronkroeker commented 4 months ago

Hi @df-sloughton,

Looks like the ruby gem docker-api 1.34.2 depends on ruby gem excon, and the excon gem was recently updated from [0.110.0](https://rubygems.org/gems/excon/versions/0.110.0 to 0.111.0 on July 8th.

[2024-08-02T06:23:46+00:00] FATAL: FrozenError: can't modify frozen Array: [:allow_unstubbed_requests, :body, :chunk_size, :debug_request, :debug_response, :dns_timeouts, :headers, :instrumentor, :logger, :method, :middlewares, :password, :path, :persistent, :pipeline, :query, :read_timeout, :request_block, :resolv_resolver, :response_block, :stubs, :timeout, :user, :versions, :write_timeout]

Based on the error above and when we compare the difference between the excon versions, we suspect this commit broke it (.freeze was added):

https://github.com/excon/excon/compare/v0.110.0...v0.111.0#diff-870d481afe6dde6869e2e7aaa8586034283a992a04f4ed683ade71b99ef7ddf3R66

In cookbooks 4.0.0 we use docker cookbook 4.12. Perhaps the docker cookbook 4.12 is modifying the array which is no longer allowed.

Can you try with cookbook 5.0.0? https://github.com/Esri/arcgis-cookbook/releases/tag/v5.0.0

I was able to get 10.9.1 Notebook Server to completely deploy (including docker) using 5.0.0. I suspect this is because we updated docker cookbook to 11.3.4, which also updated the docker-api gem to 2.3.0 which works well with excon 0.111.0.

Thanks, Cameron K.

cameronkroeker commented 3 months ago

@df-sloughton I found a workaround that will allow you to continue to use cookbooks 4.0.0.

  1. Check to see if excon 0.111.0 is installed
    • /opt/cinc/embedded/bin/gem list | grep excon
  2. If excon 0.111.0 is installed, then uninstall it
    • /opt/cinc/embedded/bin/gem uninstall excon -v 0.111.0
  3. Install excon 0.110.0
    • /opt/cinc/embedded/bin/gem install excon -v 0.110.0
  4. Check to see if docker-api 1.34.2 is installed, if not then install:
    • /opt/cinc/embedded/bin/gem list | grep docker-api
    • /opt/cinc/embedded/bin/gem install docker-api -v 1.34.2
  5. Within /etc/cinc/client.rb (if it does not exist, create it), or if using Chef (/etc/chef/client.rb) put:
    • skip_gem_metadata_installation true
  6. Run cinc/chef client to install/configure Notebook Server
cameronkroeker commented 3 months ago

@df-sloughton we have patched the arcgis-notebooks cookbook for several versions of arcgis cookbooks (3.8.0, 4.0.0, 4.1.0, and 4.2.0). So if you still need to use v4.0.0, feel free to re-download it from here:

df-sloughton commented 3 months ago

Yes the patched v4.0.0 worked well again, thanks for prompt response.

cameronkroeker commented 3 weeks ago

fixed: https://github.com/Esri/arcgis-cookbook/issues/393#issuecomment-2272189895