cannot access '/var/log/tower/profile/': Permission denied

[TheRealHaoLiu]

Please confirm the following

• [X] I agree to follow this project's code of conduct.
• [X] I have checked the current issues for duplicates.
• [X] I understand that AWX is open source software provided for free and that I might not receive a timely response.
• [X] I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

bash-4.4$ ls /var/log/tower/profile/ ls: cannot access '/var/log/tower/profile/': Permission denied

unable to access profile for debugging

AWX version

devel

Select the relevant components

• [ ] UI
• [X] API
• [ ] Docs
• [ ] Collection
• [ ] CLI
• [X] Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

exec into container and do ls /var/log/tower/profile/

Expected results

able to access '/var/log/tower/profile/'

Actual results

unable to access '/var/log/tower/profile/'

Additional information

No response

[shanemcd]

What code is trying to access that directory?

[AlanCoding]

https://github.com/ansible/awx/blob/devel/awx/main/middleware.py#L41

It's used by TimingMiddleware

but that self.dest seems to be defunct, it calls out to a utils method

https://github.com/ansible/awx/blob/devel/awx/main/utils/profiling.py#L73

I believe that's the functional place we set that location.

[shanemcd]

Is that the right place for it? We removed /var/log/tower in https://github.com/ansible/awx/commit/6d11003975948855033605624dfe9f6521c963a9. I'd hate to add it back.

[jsprhssng]

@shanemcd - I get a similar error when enabling debugging via web ui. AWX web responds 'A server error has occurred.'

Log from the web container shows;

PermissionError: [Errno 13] Permission denied: '/var/log/tower'

[AlanCoding]

^ Interesting...

https://github.com/ansible/awx/pull/14211

The error I pasted in that isn't the same as above, but it makes sense that it would be a file-does-not-exist in the awx_devel container, but a Permission denied error in OCP, with both having the same root cause. There was a change some time ago when we had OCP set the environment variable AWX_LOGGING_MODE=stdout. I'm interested to ask if you could check to see if that's absent in your case, and that might explain the problem, which may come from a mismatch of operator/AWX versions.

[prabhjotbawa]

I've no clue why the app suddenly stopped working but I'm seeing the a similar error in the awx-web pod.

Traceback (most recent call last):
78
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 55, in inner
79
response = get_response(request)
80
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/utils/deprecation.py", line 136, in __call__
81
response = self.process_response(request, response)
82
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/middleware.py", line 58, in process_response
83
response['X-API-Profile-File'] = self.prof.stop()
84
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/profiling.py", line 129, in stop
85
res = self.output_results()
86
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/profiling.py", line 90, in output_results
87
super().output_results()
88
File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/profiling.py", line 22, in output_results
89
os.makedirs(self.dest)
90
File "/usr/lib64/python3.9/os.py", line 215, in makedirs
91
makedirs(head, exist_ok=exist_ok)
92
File "/usr/lib64/python3.9/os.py", line 225, in makedirs
93
mkdir(name, mode)
94
PermissionError: [Errno 13] Permission denied: '/var/log/tower'
95
2023-07-07 17:28:13,533 ERROR [686d5b6a4fa04b478fc50be5ea43977e] django.request Internal Server Error: /

I added AWX_LOGGING_MODE as an environment variable but still see the same error, I've installed the application via the awx operator 2.3.0

[prabhjotbawa]

I think there is a bug in the Troubleshooting settings page. At least that was the path that I hit and got stuck with the above error.

I edited the page to set "Debug Web requests" as ON but then started seeing the error

I removed the database records for keys AWX_CLEANUP_PATHS, AWX_REQUEST_PROFILE and RECEPTOR_RELEASE_WORK, cleared the web cache and was able to login again.

[jsprhssng]

I think there is a bug in the Troubleshooting settings page. At least that was the path that I hit and got stuck with the above error.

I edited the page to set "Debug Web requests" as ON but then started seeing the error

I removed the database records for keys AWX_CLEANUP_PATHS, AWX_REQUEST_PROFILE and RECEPTOR_RELEASE_WORK, cleared the web cache and was able to login again.

Yes i think its a bug too, we had the same error, reported it here https://github.com/ansible/awx-operator/issues/1485

[janorn]

Just a short help for us non sql literal Run this sql on the awx db:

DELETE FROM conf_setting
WHERE key = 'AWX_CLEANUP_PATHS';
DELETE FROM conf_setting
WHERE key = 'AWX_REQUEST_PROFILE';
DELETE FROM conf_setting
WHERE key = 'RECEPTOR_RELEASE_WORK';

Thanks to @prabhjotbawa for the hint!

[prabhjotbawa]

@janorn Thanks for the mention, and also sharing the command to delete those entries, I forgot to mention the name of the table :). We could also remove them in a single command:

DELETE from conf_setting where key in ('AWX_CLEANUP_PATHS','AWX_REQUEST_PROFILE','RECEPTOR_RELEASE_WORK');