[BUG] pulp-worker crashing when Pulp installed with object storage

[git-hyagi]

In OCP environments (not tested in minikube yet), deploying Pulp with object storage makes pulp-worker pods get into a crashloobackoff state. Checking the logs it seems a lack of permission:

Traceback (most recent call last):
  File "/usr/local/bin/pulpcore-worker", line 8, in <module>
    sys.exit(worker())
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pulpcore/tasking/entrypoint.py", line 28, in worker
    NewPulpWorker().run_forever()
  File "/usr/local/lib/python3.10/site-packages/pulpcore/tasking/pulpcore_worker.py", line 378, in run_forever
    with WorkerDirectory(self.name):
  File "/usr/local/lib/python3.10/site-packages/pulpcore/tasking/storage.py", line 83, in __enter__
    self.create()
  File "/usr/local/lib/python3.10/site-packages/pulpcore/tasking/storage.py", line 133, in create
    super().create()
  File "/usr/local/lib/python3.10/site-packages/pulpcore/tasking/storage.py", line 40, in create
    os.makedirs(self.path, mode=self.MODE)
  File "/usr/lib64/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/lib/pulp/tmp/14@pulp-worker-5f557db496-dl7d8'

sh-5.1$ ls -ld /var/lib/pulp/tmp/
drwxr-xr-x. 2 700 root 6 Oct  5 04:15 /var/lib/pulp/tmp/
sh-5.1$ id
uid=1000830000(1000830000) gid=0(root) groups=0(root),1000830000

[fao89]

what do you mean by "with object storage"? file object storage? or s3/azure object storage?

[git-hyagi]

what do you mean by "with object storage"?

Hum ... storage like AWS S3, Azure Blob, or Google Cloud Storage.

file object storage? or s3/azure object storage?

Not sure if I understood your question regarding "file object storage" ... you mean file storage or object storage? https://cloud.google.com/learn/what-is-object-storage#section-4 https://docs.openshift.com/container-platform/4.10/scalability_and_performance/optimizing-storage.html#available-persistent-storage-options_persistent-storage

[fao89]

I meant: https://github.com/pulp/pulp-operator/blob/main/bundle/manifests/pulp-operator.clusterserviceversion.yaml#L337-L341

[git-hyagi]

https://github.com/pulp/pulp-oci-images/pull/336