"No such file or directory - aws" [arcgis-repository::s3files2]

[travisnburton]

Unable to execute awscli commands via recipe arcgis-repository::s3files2

Cinc output below, followed by verification of aws binary and $PATH for arcgis user ("run_as_user" : "arcgis")

Starting Cinc Client, version 15.17.4
resolving cookbooks for run list: ["arcgis-repository::s3files2"]
Synchronizing Cookbooks:
  - arcgis-repository (3.8.0)
  - s3_file (2.8.5)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 3 resources
Recipe: arcgis-repository::s3files2
  * directory[/opt/software/archives] action create (up to date)
  * execute[Download Portal_for_ArcGIS_Linux_1091_180199.tar.gz] action run

    ================================================================================
    Error executing action `run` on resource 'execute[Download Portal_for_ArcGIS_Linux_1091_180199.tar.gz]'
    ================================================================================

    Errno::ENOENT
    -------------
    No such file or directory - aws

    Resource Declaration:
    ---------------------
    # In /root/.cinc/local-mode-cache/cache/cookbooks/arcgis-repository/recipes/s3files2.rb

     54:     execute "Download #{filename}" do
     55:       command "aws s3 cp s3://#{s3_bucket}/#{s3_key} #{path} --region #{s3_region} --no-progress"
     56:       environment keys
     57:       not_if { ::File.exist?(::File.join(node['arcgis']['repository']['local_archives'], filename)) }
     58:     end
     59:   end

    Compiled Resource:
    ------------------
    # Declared in /root/.cinc/local-mode-cache/cache/cookbooks/arcgis-repository/recipes/s3files2.rb:54:in `block in from_file'

    execute("Download Portal_for_ArcGIS_Linux_1091_180199.tar.gz") do
      action [:run]
      default_guard_interpreter :execute
      command "aws s3 cp s3://arcgisstore-us-east-1/10.9.1/setups/Portal_for_ArcGIS_Linux_1091_180199.tar.gz /opt/software/archives/Portal_for_ArcGIS_Linux_1091_180199.tar.gz --region us-east-1 --no-progress"
      backup 5
      declared_type :execute
      cookbook_name "arcgis-repository"
      recipe_name "s3files2"
      environment {}
      domain nil
      user nil
      not_if { #code block }
    end

    System Info:
    ------------
    chef_version=15.17.4
    platform=redhat
    platform_version=8.6
    ruby=ruby 2.6.7p197 (2021-04-05 revision 67941) [x86_64-linux]
    program_name=/opt/cinc/bin/cinc-client
    executable=/opt/cinc/bin/cinc-client

Running handlers:
[2022-06-30T20:10:06+00:00] ERROR: Running exception handlers
Running handlers complete
[2022-06-30T20:10:06+00:00] ERROR: Exception handlers complete
Cinc Client failed. 0 resources updated in 01 seconds
[2022-06-30T20:10:06+00:00] FATAL: Stacktrace dumped to /root/.cinc/local-mode-cache/cache/cinc-stacktrace.out
[2022-06-30T20:10:06+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2022-06-30T20:10:06+00:00] FATAL: Errno::ENOENT: execute[Download Portal_for_ArcGIS_Linux_1091_180199.tar.gz] (arcgis-repository::s3files2 line 54) had an error: Errno::ENOENT: No such file or directory - aws

root@portal-2 ~]# which aws
/usr/local/bin/aws
[root@portal-2 ~]# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

[root@portal-2 ~]# su arcgis
[arcgis@portal-2 root]$ echo $PATH
/home/arcgis/.local/bin:/home/arcgis/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
[arcgis@portal-2 root]$ which aws
/usr/local/bin/aws

[cameronkroeker]

Hi @travisnburton,

Which version of aws cli is installed?

aws --version

Does running the following command manually work?

aws s3 cp s3://arcgisstore-us-east-1/10.9.1/setups/Portal_for_ArcGIS_Linux_1091_180199.tar.gz /opt/software/archives/Portal_for_ArcGIS_Linux_1091_180199.tar.gz --region us-east-1

Also, which OS Platform and version is the EC2 instance, Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, or RHEL?

Thanks, Cameron K.

[travisnburton]

[root@portal-1 ~]# aws --version
aws-cli/2.7.12 Python/3.9.11 Linux/4.18.0-372.9.1.el8.x86_64 exe/x86_64.rhel.8 prompt/off

[root@portal-1 ~]# aws s3 cp s3://arcgisstore-us-east-1/10.9.1/setups/Portal_for_ArcGIS_Linux_1091_180199.tar.gz /opt/software/archives/Portal_for_ArcGIS_Linux_1091_180199.tar.gz --region us-east-1
Completed 1.4 GiB/3.8 GiB (207.2 MiB/s) with 1 file(s) remaining

    System Info:
    ------------
    chef_version=15.17.4
    platform=redhat
    platform_version=8.6
    ruby=ruby 2.6.7p197 (2021-04-05 revision 67941) [x86_64-linux]
    program_name=/opt/cinc/bin/cinc-client
    executable=/opt/cinc/bin/cinc-client

[cameronkroeker]

Thanks @travisnburton for sharing those details. Are you running ciic-client as root, or as the ec2-user (or other user) with sudo?

Thanks, Cameron K

[travisnburton]

Hi @cameronkroeker - cinc-client is being executed from an SSM document:

    {
      "action": "aws:runShellScript",
      "name": "DownloadSoftwareAndLicense",
      "inputs": {
        "runCommand": [
          "aws s3 cp s3://{{ S3Bucket}}/{{ ServerType }}/cinc-s3files.json /opt/cinc/s3files.json",
          "aws s3 cp s3://{{ S3Bucket}}/common/license.json /opt/software/licenses/license.json",
          "sudo /opt/cinc/bin/cinc-client -z -j /opt/cinc/s3files.json --config-option \"cookbook_path=/opt/cinc/cookbooks\""
        ]
      }
    },

So the client is executed by user ssm-agent with sudo.

I assumed the failure was with the arcgis user, which is defined in the client configuration file "run_as_user" : "arcgis".

However, your comment made me turn to sudo, which is where the problem lies.

sh-4.4$ whoami
ssm-user
sh-4.4$ sudo aws --version
sudo: aws: command not found
sh-4.4$ sudo echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
sh-4.4$ sudo which aws
which: no aws in (/sbin:/bin:/usr/sbin:/usr/bin)

Symlinking to /usr/bin solved the issue:

sh-4.4$ sudo ln -s /usr/local/bin/aws /usr/bin/aws
sh-4.4$ sudo which aws
/bin/aws

sh-4.4$ sudo /opt/cinc/bin/cinc-client -z -j /opt/cinc/s3files.json --config-option cookbook_path=/opt/cinc/cookbooks
[2022-06-30T21:38:39+00:00] WARN: No config file found or specified on command line. Using command line options instead.
Starting Cinc Client, version 15.17.4
resolving cookbooks for run list: ["arcgis-repository::s3files2"]
Synchronizing Cookbooks:
  - arcgis-repository (3.8.0)
  - s3_file (2.8.5)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 3 resources
Recipe: arcgis-repository::s3files2
  * directory[/opt/software/archives] action create (up to date)
  * execute[Download Portal_for_ArcGIS_Linux_1091_180199.tar.gz] action run
  ....

Thanks again for all your help!

[cameronkroeker]

Ah ha! Great find @travisnburton. Glad to hear the issue is resolved by symlinking /usr/bin, thanks for sharing. Yeah the setups are not downloaded as "run_as_user" because in theory this account may not yet exist (gets created in the system recipe which is not defined in the s3files). But definitely see how that is confusing.

Happy Automating, Cameron K.