Notebook Server Recipes fail in Disconnected Environment

jmturco17 commented 4 months ago

I'm attempting to deploy ArcGIS Enterprise with an ArcGIS Server Notebook role in a air-gapped (disconnected) environment. The Notebook server chef-solo run appears to be reaching out to "index.rubygems.org" which fails. Details on the issues can be found below:

FATAL: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '17' -------Begin output of ['bundle', 'install'] ------ STDOUT: STDERR: Could not reach host index.rubgems.org. Check your network connection and try again -------End output of ["bundle", "install"] Ran ["bundle", "install"] returned 17

jmturco17 commented 4 months ago

CInc Version is 17.0.0

cameronkroeker commented 4 months ago

CInc Version is 17.0.0

Thanks @jmturco17. Can you also give Cinc client 18 a try? Curious to see if the issue is reproducible with the latest version of Cinc client.

cameronkroeker commented 4 months ago

I suspect the issue could be related to the third party docker cookbook that the notebook server cookbook uses to install docker engine. This looks like it could be related:

https://github.com/sous-chefs/docker/issues/682

Additionally, there is this requirement of a network accessible web server containing the docker binaries:

https://github.com/sous-chefs/docker/blob/main/README.md#requirements

Might need to install/configure docker using a different method and set the following attribute in the json config file to false:

node['arcgis']['notebook_server']['install_docker'] = If set to true, the arcgis-notebooks::docker recipe installs the Docker engine. Default value is false for RHEL Linux and true otherwise.

https://github.com/Esri/arcgis-cookbook/tree/main/cookbooks/arcgis-notebooks#attributes

jmturco17 commented 4 months ago

I'm going to be trying out CINC 18.4.12 next week to give it a try.

I'm manually installing docker before even getting to the Notebook Server installation. I've also removed the docker install fro the run-list and the issues persists.

cameronkroeker commented 4 months ago

I'm going to be trying out CINC 18.4.12 next week to give it a try.

I'm manually installing docker before even getting to the Notebook Server installation. I've also removed the docker install fro the run-list and the issues persists.

Please share your recipe run-list so we can get a better picture of what all is being ran. If possible your json config file as well with sensitive information such as passwords/machine names removed.

jmturco17 commented 4 months ago

Here is the whole JSON file minues the sensitive data

{
    "arcgis": {
        "version": "<version>",
        "run_as_user": "arcgis",
        "repository": {
            "archives": "/opt/software/archives",
            "setups": "/opt/software/setups"
        },
        "notebook_server": {
            "install_dir": "/opt",
            "install_system_requirements": true,
            "install_samples_data": true,
            "authorization_file": "/opt/software/authorization_files/<version>/notebook_advanced_authorization.eslf",
            "license_level": "standard",
            "admin_username": "<ags_username>",
            "admin_password": "<ags_password>",
            "directories_root": "/gisdata/notebookserver/directories",
            "workspace": "/gisdata/notebookserver/directories/arcgisworkspace",
            "config_store_type": "FILESYSTEM",
            "config_store_connection_string": "/gisdata/notebookserver/config-store",      
            "log_level": "WARNING",
            "log_dir": "/opt/arcgis/notebookserver/usr/logs",
            "configure_autostart": true,
            "system_properties": { }
        }
    },
    "run_list": [
        "recipe[arcgis-enterprise::system]",
        "recipe[arcgis-notebooks::docker]",
        "recipe[arcgis-notebooks::iptables]",
        "recipe[arcgis-notebooks::restart_docker]",
        "recipe[arcgis-notebooks::server]"
    ]
}

Also, please note, I've tried to slim this down be removing docker and system from the run_list and still received the issue.

jmturco17 commented 3 months ago

I can also confirm that upgrading the the latest chef recipe book DOES NOT resolve the issue.

cameronkroeker commented 3 months ago

Thanks @jmturco17 for the additional information. I dug into this a bit and discovered the following gems are trying to be installed from the internet:

As a workaround those can be downloaded on a machine that has internet access and copied over to the air-gapped environment, then installed using the following commands:

/opt/cinc/embedded/bin/gem install --local /tmp/excon-0.110.0.gem

/opt/cinc/embedded/bin/gem install --local /tmp/docker-api-2.2.0.gem

Next you will want to add skip_gem_metadata_installation to the /etc/cinc/client.rb file (or if using chef /etc/chef/client.rb file) and set it to true so that it skips installing gems from the internet:

# The client.rb file specifies how Chef Infra Client is configured on a node
# See https://docs.chef.io/config_rb_client/ for detailed configuration options
#
# Minimal example configuration:
# node_name  "THIS_NODE_NAME"
# chef_server_url  "https://CHEF.MYCOMPANY.COM/organizations/MY_CHEF_ORG"
# chef_license  "accept"
file_cache_path "/var/cinc/cache"
gem_installer_bundler_options "--local"
skip_gem_metadata_installation true

For more information about the Chef client.rb: https://docs.chef.io/config_rb_client/.

cameronkroeker commented 3 months ago

Hi @jmturco17,

Here is a more detailed summary of using the following recipes in a disconnected environment.

"run_list": [
        "recipe[arcgis-enterprise::system]",
        "recipe[arcgis-notebooks::docker]",
        "recipe[arcgis-notebooks::iptables]",
        "recipe[arcgis-notebooks::restart_docker]",
        "recipe[arcgis-notebooks::server]"
]

arcgis-enterprise::system

https://github.com/Esri/arcgis-cookbook/tree/main/cookbooks/arcgis-enterprise#system

Installs ruby gem multipart-post from internet
- https://rubygems.org/gems/multipart-post/versions/2.4.1
- To install locally:
  - /opt/cinc/embedded/bin/gem install --local /home/ubuntu/multipart-post-2.4.1.gem
- To prevent Cinc/Chef client from installing missing gems from internet, in /etc/cinc/client.rb, or if using chef /etc/chef/client.rb, set skip_gem_metadata_installation true
  - https://docs.chef.io/config_rb_client/
Depending on OS platform it will install missing packages from the internet, ensure these are installed so that they will be skipped. The list of packages can be found here:

https://github.com/Esri/arcgis-cookbook/blob/c92c7768637d5e5e2bb633a867107b8e9c206953/cookbooks/arcgis-enterprise/attributes/default.rb#L60-L78

arcgis-notebooks::docker

https://github.com/Esri/arcgis-cookbook/tree/main/cookbooks/arcgis-notebooks#docker

Requires the following 2 ruby gems
- https://rubygems.org/gems/excon
- https://rubygems.org/gems/docker-api/versions/2.2.0
- To install locally:
  - /opt/cinc/embedded/bin/gem install --local /home/ubuntu/excon-0.110.0.gem
  - /opt/cinc/embedded/bin/gem install --local /home/ubuntu/docker-api-2.2.0.gem
  - To prevent Cinc/Chef client from installing missing gems from internet, in /etc/cinc/client.rb, or if using chef /etc/chef/client.rb, set skip_gem_metadata_installation true
  - https://docs.chef.io/config_rb_client/
Downloads remote_file docker binaries (https___download_docker_com_linux_ubuntu_gpg)
- There isn't a straight forward workaround here other than perhaps hosting it localling
  - https://github.com/sous-chefs/docker/blob/main/README.md#requirements
  - I suggest setting node['arcgis']['notebook_server']['install_docker'] to false in your json config so that it skips downloading/installing docker. Docker will need to be installed/configured outside of Chef prior.

arcgis-notebooks::iptables

Requires iptables package
To prevent Cinc/Chef client from installing missing gems from internet, in /etc/cinc/client.rb, or if using chef /etc/chef/client.rb, set skip_gem_metadata_installation true
- https://docs.chef.io/config_rb_client/

arcgis-notebooks::restart_docker

https://github.com/Esri/arcgis-cookbook/tree/main/cookbooks/arcgis-notebooks#restart_docker

To prevent Cinc/Chef client from installing missing gems from internet, in /etc/cinc/client.rb, or if using chef /etc/chef/client.rb, set skip_gem_metadata_installation true
- https://docs.chef.io/config_rb_client/

recipe[arcgis-notebooks::server]

https://github.com/Esri/arcgis-cookbook/tree/main/cookbooks/arcgis-notebooks#server

To prevent Cinc/Chef client from installing missing gems from internet, in /etc/cinc/client.rb, or if using chef /etc/chef/client.rb, set skip_gem_metadata_installation true
- https://docs.chef.io/config_rb_client/
Requires the setup archive installation files
- https://esri.github.io/arcgis-cookbook/templates/arcgis-notebook-server/11.3.html#required-arcgis-software-repository-content
Requires either advanced or standard notebook server license file (either .ecp or .ESLF)

When deploying in cloud environments such as AWS, it might be best or easiest to first create an image (AMI) with the gem files, linux packages, docker and chef/cinc client all installed. Then use that AMI in the disconnected environment.

Thanks, Cameron K.

jmturco17 commented 3 months ago

Thanks Cameron, I saw your post earlier and have been working to get that running in the last day or so. I'll let you know how it all looks hopefully early next week once I'm able to test again.

cameronkroeker commented 3 months ago

@jmturco17 I should also mention, there is an issue with using a password protected ESLF with arcgis-notebooks cookbook. It's missing the authorization_options attribute which is used for passing in the password of the ESLF. Here are a couple workarounds:

Workaround 1: Simplest, but requires hardcoding password in cookbook which is not ideal from a security perspective.

Modify:

https://github.com/Esri/arcgis-cookbook/blob/c92c7768637d5e5e2bb633a867107b8e9c206953/cookbooks/arcgis-notebooks/resources/server.rb#L316

To: args = "-f \"#{@new_resource.authorization_file}\" -p \"yourpassword\""

Workaround 2: Recommended approach, but requires more changes than workaround 1.

Note: This is the fix that will be used in a future release of the cookbooks. The same issue is also in arcgis-video, arcgis-geoevent, arcgis-workflow-manager, and arcgis-mission cookbooks.

1. Introduce new authorization_options attribute

In: https://github.com/Esri/arcgis-cookbook/blob/c92c7768637d5e5e2bb633a867107b8e9c206953/cookbooks/arcgis-notebooks/resources/server.rb#L26

Add:

attribute :authorization_options, :kind_of => String

2. Pass in the authorization_options attribute in args

Change:

https://github.com/Esri/arcgis-cookbook/blob/c92c7768637d5e5e2bb633a867107b8e9c206953/cookbooks/arcgis-notebooks/resources/server.rb#L316

To:

args = "-f \"#{@new_resource.authorization_file}\" #{@new_resource.authorization_options}"

3. Modify server and server_node recipes to use the attribute:

Add:

authorization_options node['arcgis']['notebook_server']['authorization_options']

For example:

arcgis_notebooks_server 'Authorize ArcGIS Notebook Server' do
  authorization_file node['arcgis']['notebook_server']['authorization_file']
  authorization_file_version node['arcgis']['notebook_server']['authorization_file_version']
  authorization_options node['arcgis']['notebook_server']['authorization_options']
  retries 2
  retry_delay 30
  notifies :stop, 'arcgis_notebooks_server[Stop ArcGIS Notebook Server]', :immediately
  action :authorize
end

4. Add and Set default value for authorization_options to empty string.

In: https://github.com/Esri/arcgis-cookbook/blob/main/cookbooks/arcgis-notebooks/attributes/default.rb

Add: notebook_server['authorization_options'] = ""

5. Define authorization_options attribute in your json configuration file.

"notebook_server": {
   "install_dir": "/opt",
   "install_system_requirements": true,
   "install_samples_data": true,
   "authorization_file": "/opt/software/authorization_files/<version>/notebook_advanced_authorization.eslf",
   "authorization_options": "-p theESLFPassword",
   "license_level": "standard",
   "admin_username": "<ags_username>",
   "admin_password": "<ags_password>",
   "directories_root": "/gisdata/notebookserver/directories",
   "workspace": "/gisdata/notebookserver/directories/arcgisworkspace",
   "config_store_type": "FILESYSTEM",
   "config_store_connection_string": "/gisdata/notebookserver/config-store",      
   "log_level": "WARNING",
   "log_dir": "/opt/arcgis/notebookserver/usr/logs",
   "configure_autostart": true,
   "system_properties": { }
}

jmturco17 commented 3 months ago

I made the changes in the /etc/cinc/client.rb file and cinc looks like it's still reaching out

cameronkroeker commented 3 months ago

skip_gem_metadata_installation true

You can pass it in directly via the command line as well, which overrides the /etc/cinc/client.rb. For example:

cinc-client -z -j your.json --config-option "skip_gem_metadata_installation=true"

or

cinc-solo -j your.json --config-option "skip_gem_metadata_installation=true"

jmturco17 commented 3 months ago

Thanks Cameron! The command line option seemed to take, working on the rest of the installation steps now!

Esri / arcgis-cookbook