Unable to create `/opt/pdc_discovery` on a new VM

[Beck-Davis]

Expected behavior

Deploy would complete from ansible-tower

Actual behavior

On a brand new VM deploying from ansible-tower fails with the following error: TASK [cap deploy] ************************************************************** fatal: [towerdeploy1.princeton.edu]: FAILED! => {"changed": true, "cmd": "cd /tmp/pdc_discovery && BRANCH=main /usr/local/bin/cap staging deploy", "delta": "0:00:02.528361", "end": "2024-07-12 20:21:38.642261", "msg": "non-zero return code", "rc": 1, "start": "2024-07-12 20:21:36.113900", "stderr": "#<Thread:0x00007f38624e8ad0 /usr/local/lib/ruby/gems/3.2.0/gems/sshkit-1.23.0/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):\n/usr/local/lib/ruby/gems/3.2.0/gems/sshkit-1.23.0/lib/sshkit/runners/parallel.rb:15:inrescue in block (2 levels) in execute': Exception while executing as deploy@pdc-discovery-staging1.princeton.edu: mkdir exit status: 1 (SSHKit::Runner::ExecuteError)\nmkdir stdout: Nothing written\nmkdir stderr: mkdir: cannot create directory ‘/opt/pdc_discovery’: Permission denied\nmkdir: cannot create directory ‘/opt/pdc_discovery’: Permission denied\n\n\tfrom /usr/local/lib/ruby/gems/3.2.0/gems/sshkit-1.23.0/lib/sshkit/runners/parallel.rb:11:in …`

Specifically it fails to create /opt/pdc_discovery

Steps to replicate

Create a new vm. Run deploy rails project from ansible-towe

Impact of this bug

Cannot deploy from ansible-tower

Honeybadger link and code snippet, if applicable

Implementation notes, if any

[acozine]

Did you run the pdc_discovery provisioning playbook against the VM before deploying? My guess is that the directory would get created as part of the rails_app role.

[jrgriffiniii]

I could not reproduce this, as the princeton_ansible Playbook seems to ensure that the /opt directory and subdirectories are created with the following:

TASK [roles/rails_app : rails_app | Create app directory structure] ************************************************************************************
ok: [pdc-discovery-staging1.princeton.edu] => (item=pdc_discovery)
ok: [pdc-discovery-staging1.princeton.edu] => (item=pdc_discovery/shared)
ok: [pdc-discovery-staging1.princeton.edu] => (item=pdc_discovery/shared/tmp)

Please see https://github.com/pulibrary/princeton_ansible/blob/main/roles/rails_app/tasks/main.yml#L37

[jrgriffiniii]

I apologize, I misunderstood the acceptance criteria and did not realize that I must request to create an entirely new VM in order to reproduce this (I had assumed that pdc-discovery-staging1 was replaced with a new VM, and that this needed to be fixed on pdc-discovery-staging1.

[jrgriffiniii]

I am labeling this as blocked as this shall require the assistance of at least one member of the Operations Team.

[carolyncole]

I can verify that a brand new VM gets this error if the playbook has not been run on it. BUT after the playbook run this error does not occur.