permission denied errors in container logs for splunk

[aneesh786]

Discussed in https://github.com/splunk/splunk-operator/discussions/1252

^{Originally posted by **aneesh786** November 10, 2023} Hi Team, Iam trying to install a cluster manager by using spark operator 2.4.0. Containers are failing with below error. STDOUT: Updating /opt/splunk/etc STDERR: tar: anonymizer: Cannot mkdir: Permission denied tar: anonymizer: Cannot mkdir: Permission denied tar: anonymizer/anonymizer-time.ini: Cannot open: No such file or directory tar: anonymizer: Cannot mkdir: Permission denied tar: anonymizer/dictionary.txt: Cannot open: No such file or directory tar: anonymizer: Cannot mkdir: Permission denied tar: anonymizer/names.txt: Cannot open: No such file or directory tar: anonymizer: Cannot mkdir: Permission denied tar: anonymizer/private-terms.txt: Cannot open: No such file or directory tar: anonymizer: Cannot mkdir: Permission denied tar: anonymizer/public-terms.txt: Cannot open: No such file or directory tar: apps: Cannot mkdir: Permission denied tar: apps: Cannot mkdir: Permission denied tar: apps/alert_logevent: Cannot mkdir: No such file or directory tar: apps: Cannot mkdir: Permission denied tar: system/static/moreAppsIcon.png: Cannot open: No such file or directory tar: system: Cannot mkdir: Permission denied tar: system/static/splunkrc_cmds.xml: Cannot open: No such file or directory tar: users: Cannot mkdir: Permission denied tar: users: Cannot mkdir: Permission denied tar: users/users.ini.default: Cannot open: No such file or directory tar: Exiting with failure status due to previous errors also looked at the container and looks like /opt/splunk/etc does n't have permission. [splunk@splunk-cm-cluster-manager-0 splunk]$ ls -lrt total 3220 -r--r--r--. 1 splunk splunk 85405 Aug 25 17:02 license-eula.txt -r--r--r--. 1 splunk splunk 57 Aug 25 17:02 copyright.txt -r--r--r--. 1 splunk splunk 520 Aug 25 17:05 README-splunk.txt drwxr-xr-x. 3 splunk splunk 45 Aug 25 17:24 include drwxr-xr-x. 2 splunk splunk 66 Aug 25 17:24 cmake drwxr-xr-x. 2 splunk splunk 55 Aug 25 17:24 quarantined_files drwxr-xr-x. 3 splunk splunk 58 Aug 25 17:24 openssl -rw-r--r--. 1 splunk splunk 0 Aug 25 17:24 ftr drwxr-xr-x. 2 splunk splunk 54 Aug 25 17:24 swidtag -r--r--r--. 1 splunk splunk 3200489 Aug 25 17:25 splunk-9.1.1-64e843ea36b1-linux-2.6-x86_64-manifest drwxr-xr-x. 1 splunk splunk 20 Sep 8 18:14 share drwxr-xr-x. 1 splunk splunk 43 Sep 8 18:14 lib drwxr-xr-x. 1 splunk splunk 169 Sep 8 18:14 bin drwxrwxrwx. 2 root root 6 Nov 9 18:39 var drwxrwxrwx. 2 root root 6 Nov 9 18:39 etc [splunk@splunk-cm-cluster-manager-0 splunk]$ drwxrwxrwx. 2 root root 6 Nov 10 07:20 etc [splunk@splunk-cm-cluster-manager-0 splunk]$ cd etc [splunk@splunk-cm-cluster-manager-0 etc]$ ls ls: cannot open directory '.': Permission denied I tried standalone deployment and that too failed with same error. Do we have any fix for this ?

[vivekr-splunk]

@aneesh786. can you help us investigate this issue further, which platform are you trying this. which storage class are you trying it on. thank you

[aneesh786]

@vivekr-splunk While trying to create cluster manager or any resource, my container was unable to mount /etc and /var. I had configured persistent storage using local-path. I believe this was the problem. Pod was unable to create volumes with this storage. Created another storage class using ceph cluster and made it as default storage class. This resolved the issue. I was using K8S cluster to test splunk

[vivekr-splunk]

closing issue based on customer response.