Sync doesn't work with subdirectories

[bribroder]

What happened?

When devspace dev runs, during the initial sync I get a permissions error from inotify when it tries to descend into subdirectories of my container's working dir (/opt/app):

$ devspace dev
info Using namespace 'asdf'
info Using kube context 'arn:aws:eks:us-west-1:123:cluster/asdf'
deploy:app Applying manifests with kubectl...
deploy:app deployment.apps/asdf configured
deploy:app secret/asdf-secrets configured
deploy:app Successfully deployed app with kubectl
dev:app Waiting for pod to become ready...
dev:app DevSpace is waiting, because Pod asdf-devspace-66bfddbf5b-tvlfg has status: ContainerCreating
dev:app Selected pod asdf-devspace-66bfddbf5b-tvlfg
dev:app sync  Sync started on: ./app/ <-> /opt/app
dev:app sync  Waiting for initial sync to complete
dev:app sync  Initial sync completed
start_dev: initial sync: Sync - connection lost to pod asdf/asdf-devspace-66bfddbf5b-tvlfg: Use inotify as watching method in container
2024/03/20 15:51:44 error watching path /opt/app: lstat /opt/app/libs/__pycache__: permission denied
 command terminated with exit code 1
fatal exit status 1

I can exec into the running pod and see that all the subdirectories of my working dir and the sync path have permissions set to 644 instead of 755 like on the host, so the directories can't be executed--although all files do have the correct ownership permissions matching the container user.

I can actually fix the sync issue after the pod deploys:

pod_name=$(kubectl get pods -l app=asdf -n asdf -o jsonpath='{.items[0].metadata.name}')
kubectl exec -ti $pod_name -- find . -type d -exec chmod +X {} \; 

And then devspace dev will complete successfully and sync will work normally, until the container is killed and the initial sync needs to happen again

Note that my pod does not run as root

How can we reproduce the bug? (as minimally and precisely as possible)

Create a sync into the working directory with some subdirectories?

My devspace.yaml:

version: v2beta1
deployments:
  app:
    kubectl:
      manifests:
      - ./deploy/deployment.yml
dev:
  app:
    sync:
    - path: ./app/:/opt/app
      uploadExcludeFile: .devspaceignore

Local Environment:

• DevSpace Version: 6.3.12
• Operating System: mac
• ARCH of the OS: ARM64 Kubernetes Cluster:
• Cloud Provider: aws
• Kubernetes Version: 1.29

[bribroder]

Extra note, using an onUpload command to fix permissions doesn't work, it doesn't seem to run at all if the sync errors

sync:
- path: ./app:/opt/app
  uploadExcludeFile: .devspaceignore
  onUpload:
    exec:
    - command: |-
        find /opt/app -type d -exec chmod +X {} \;

Setting polling: true doesn't work around the issue either

dev:app sync  Initial sync completed
start_dev: initial sync: upstream: apply changes: apply creates: upload archive: after upload: rpc error: code = Unknown desc = untar all: decompress: create /opt/app/libs/asdf.py: open /opt/app/libs/asdf.py: permission denied
fatal exit status 1

[bribroder]

I tracked this down to initial permissions of the subdirectories in the docker image

[rafilkmp3]

I tracked this down to initial permissions of the subdirectories in the docker image

Hey, i'm having the same problem on a project of mine and if I might ask, can you elaborate a little further on the solution? Did you create a new docker image with wider permissions so the devspace runner can do his initial work correctly?

[siredmar]

@bribroder @rafilkmp3 I have the same issue. Anything that helps?

[siredmar]

Never mind. The problem on my side was the securityContext set pretty restrictive for my pod (what is best practice). So i needed to get rid of the securityContext of my container:

dev:
  app:
    labelSelector:
      app.kubernetes.io/name: myfancyapp
    namespace: mynamespace
    devImage: ghcr.io/loft-sh/devspace-containers/go:1.20-alpine
    patches:
    - op: remove
      path: spec.containers[0].securityContext
    sync:
      - path: ./:/app
        excludePaths:
        - vendor/

Hope this helps folks.