Sync fails with `Permission denied` when using juju 3.5.1

[przemeklal]

Bug Description

Sync fails with Permission denied when using juju 3.5.1.

To Reproduce

1. Deploy cos-configuration in juju 3.5.1 env (I tried latest/stable, latest/edge and older revisions such as 44 that worked on another env that we manage - it was the same behaviour with all versions).
2. Sync fails with the stderr: "Cloning into '/git'...\n/git/.git: Permission denied\n". Details below.

Environment

juju 3.5.1 ceph-csi microk8s v1.28.9

Relevant log output

08:26:57 E0731 08:26:56.444895     139 main.go:525] "msg"="too many failures, aborting" "error"="Run(git clone -v --no-checkout -b sitael-platino --depth 1 https://user:password@git.launchpad.net/~project/+git/repo /git): exit status 1: { stdout: "", stderr: "Cloning into '/git'...\n/git/.git: Permission denied\n" }" "failCount"=0

Additional context

Sitael-Platino MODEL(cos) jujumanage@infra-0101:~$ kubectl exec -it -n cos cos-config-0 -c git-sync -- sh
$ ls / -liah
total 11M
4615111 drwxr-xr-x   1 root root 4.0K Jul 31 08:12 .
4615111 drwxr-xr-x   1 root root 4.0K Jul 31 08:12 ..
4615309 -rwxr-xr-x   1 root root    0 Dec 17  2021 .dockerenv
4615310 drwxr-xr-x   4 root root 4.0K Mar 11  2022 LICENSES
4615355 drwxr-xr-x   2 root root 4.0K Dec 17  2021 bin
4615413 drwxr-xr-x   2 root root 4.0K Oct  3  2021 boot
4615128 drwxr-xr-x   4 root root 4.0K Jul 31 08:12 charm
      1 drwxr-xr-x   5 root root  360 Jul 31 08:12 dev
4615418 drwxr-xr-x  32 root root 4.0K Mar 11  2022 etc
    128 drwxr-xr-x   2 root root    6 Jul 31 08:12 git <- ownership issue?
4615857 -rwxr-xr-x   1 root root  11M Mar 11  2022 git-sync
4615858 drwxr-xr-x   2 root root 4.0K Oct  3  2021 home
4615859 drwxr-xr-x   4 root root 4.0K Dec 17  2021 lib
4616005 drwxr-xr-x   2 root root 4.0K Dec  1  2021 lib64
4616007 drwxr-xr-x   2 root root 4.0K Dec  1  2021 media
4616008 drwxr-xr-x   2 root root 4.0K Dec  1  2021 mnt
4616009 drwxr-xr-x   2 root root 4.0K Dec  1  2021 opt
      1 dr-xr-xr-x 328 root root    0 Jul 31 08:12 proc
4616011 drwx------   2 root root 4.0K Dec 17  2021 root
4616012 drwxr-xr-x   1 root root 4.0K Jul 31 08:12 run
4616015 drwxr-xr-x   2 root root 4.0K Dec 17  2021 sbin
4616028 drwxr-xr-x   2 root root 4.0K Dec  1  2021 srv
      1 dr-xr-xr-x  13 root root    0 Jul 31 08:12 sys
4616030 drwxrwxrwt   1 root root 4.0K Jul 31 08:27 tmp
4616032 drwxr-xr-x  10 root root 4.0K Dec  1  2021 usr
4621070 drwxr-xr-x  11 root root 4.0K Dec  1  2021 var
$ whoami
git-sync

[przemeklal]

The same charm revision works fine on juju 3.1.

[lucabello]

In the long term, we should probably have a rock for this, but for now we'll fix the permission error.

https://github.com/canonical/operator/blob/e79446071040a3f20817c7b499c40f68c3dd3c60/ops/pebble.py#L2466

[lucabello]

I cannot reproduce in any way, I've tried Juju 3.5.1 and 3.5.3, and the charm both from stable and edge. Could you try and reproduce again?

[lucabello]

Closing because I can't reproduce it in any way, but feel free to reopen if you have a reproducer!