.git broken in devcontainer

[ka-bu]

• VSCode Version: 1.89.1
• Local OS Version: windows 11 (10.0.22631 Build 22631)
• Remote OS Version: ubuntu 20.04 (FROM mcr.microsoft.com/vscode/devcontainers/base:ubuntu-20.04)
• Remote Extension/Connection Type: Dev Containers (WSL)

• Logs: devcontainer logs don't show any errors

Steps to Reproduce: (tested also with insider version)

1. start vscode with devcontainer (based on the image mentioned above with remoteUser specified (not that it makes a difference)

2. run some git commands which fail and find that e.g.

   • ls .git complains that ls: cannot access '.git/ORIG_HEAD': No such file or directory (i.e. file shows up with ??? as access rights and it's the only file in the .git folder were this happens) (this is what I see at the moment)

   • or the whole .git/index file is not accessible (which makes it look like all files should be deleted :))

     and every time everything works perfectly fine with git bash outside of the container.

     Also I just noticed that the docker extension claims "failed to connect. Is docker installed?" though it is installed and running otherwise the container wouldn't. (Could be a separate issue because there are regularly other issues with docker itself).

The issue started about two weeks ago (with a setup that was working for a long time before). Some days it just seemed to disappear (e.g. last Thursday/Friday it worked fine until a restart on Friday). I've tried some old extension versions and old vscode versions and also tried vscode insider version.

Does this issue occur when you try this locally?: Yes/No Does this issue occur when you try this locally and all extensions are disabled?: Yes/No

[chrmarti]

What do you get for ls -la in the container in the workspace folder? This sounds like a permission problem.

[ka-bu]

(run) (base) vscode ➜ /workspaces $ ls -la
total 8
drwxr-xr-x 3 root   root   4096 May 27 09:50 .
drwxr-xr-x 1 root   root   4096 May 27 09:50 ..
drwxrwxrwx 1 vscode vscode 4096 May 27 09:03 Recommendations

and maybe more important in the .git folder inside that Recommendations folder, I get

(run) (base) vscode ➜ /workspaces/Recommendations/.git  $ ls -lah
ls: cannot access 'index': No such file or directory
total 48K
drwxrwxrwx 1 vscode vscode 4.0K May 28 13:08 .
drwxrwxrwx 1 vscode vscode 4.0K May 27 09:03 ..
-rwxrwxrwx 1 vscode vscode    6 May 24 13:58 COMMIT_EDITMSG
-rwxrwxrwx 1 vscode vscode  507 May 24 13:55 config
-rwxrwxrwx 1 vscode vscode   73 May 21 08:38 description
-rw-r--r-- 1 vscode vscode  31K May 27 09:02 FETCH_HEAD
-rwxrwxrwx 1 vscode vscode   70 May 24 12:56 HEAD
drwxrwxrwx 1 vscode vscode 4.0K May 27 09:50 hooks
-????????? ? ?      ?         ?            ? index
drwxrwxrwx 1 vscode vscode 4.0K May 21 08:38 info
drwxrwxrwx 1 vscode vscode 4.0K May 21 08:38 logs
drwxrwxrwx 1 vscode vscode 4.0K May 27 09:38 objects
-rw-r--r-- 1 vscode vscode   41 May 27 08:11 ORIG_HEAD
-rwxrwxrwx 1 vscode vscode  16K May 21 08:38 packed-refs
drwxrwxrwx 1 vscode vscode 4.0K May 21 08:38 refs

but the above looked fine (i.e. no "???.." but "vscode vscode") after restart but immediately changed to above after the first git status.

Also a colleague for whom the setup is broken as well sees: and for him the whole setup works fine with docker 4.29.0 (while for me it doesn't)

[chrmarti]

Also a colleague for whom the setup is broken as well sees: and for him the whole setup works fine with docker 4.29.0 (while for me it doesn't)

This looks like the user is root. Does using "remoteUser": "root" change anything on your machine?

[ka-bu]

Hard to tell right away because, as mentioned above, problems seem to appear and disappear randomly and it's slightly different every time (e.g. this morning I was able to run some git commands plus a pull until when I pulled again I got a "did not sent all necessary objects"). That said, I've rebuilt with the setting and the issue disappeared for now, and I've also asked a colleague to test this. I'll update this thread in about 2 days (because last time it was randomly working and breaking again after 2 days).

[ka-bu]

This looks like the user is root

We're all using the same .devcontainer.json so he should have (had) the same remoteUser setting

[ka-bu]

Worked throughout the day, only problem is that pip behaves weird now because it has some .cache directory in /home/vscode which can't be read, so I guess I should rework the dockerfile (if this is the final solution?)

[ka-bu]

@chrmarti I can confirm now that setting the remoteUser to "root" does not solve the problem. Didn't do anything except starting the laptop, and now in the .git folder everything except HEAD is assigned to root, while HEAD is assigned to root, and again it looks like mounting index failed (i.e. ???? / "No such file or directory")

[chrmarti]

Does git work on your local machine in that workspace folder? This might be an issue with how Docker Desktop mounts the folder from the Windows filesystem. Also check the permissions on the index folder on the Windows side, maybe these can't be mapped to Linux permissions for some reason?

Please append the Dev Containers log from when you are connected to the dev container, maybe that has a hint on what is going on. (F1 > Dev Containers: Show Container Log)

[ka-bu]

Yes it appears as if it's failing to mount single files at random. As already mentioned above, everything works fine outside the container and index (is a file and) permissions look okay on windows side.

Here's the dev container log.txt where I currently see:

(run) (base) root ➜ /workspaces/Recommendations/.git $ ls -lah
ls: cannot access 'index': No such file or directory
total 52K
drwxrwxrwx 1 vscode vscode 4.0K May 30 12:24 .
drwxrwxrwx 1 vscode vscode 4.0K May 27 09:03 ..
-rwxrwxrwx 1 vscode vscode   13 May 29 16:02 COMMIT_EDITMSG
-rwxrwxrwx 1 root   root    737 May 29 15:59 config
-rwxrwxrwx 1 vscode vscode   73 May 21 08:38 description
-rw-r--r-- 1 vscode vscode  31K May 30 08:32 FETCH_HEAD
-rw-r--r-- 1 root   root     23 May 30 08:32 HEAD
drwxrwxrwx 1 vscode vscode 4.0K May 29 09:00 hooks
-????????? ? ?      ?         ?            ? index
drwxrwxrwx 1 vscode vscode 4.0K May 21 08:38 info
drwxrwxrwx 1 vscode vscode 4.0K May 21 08:38 logs
drwxrwxrwx 1 vscode vscode 4.0K May 30 08:32 objects
-rw-r--r-- 1 root   root     41 May 30 08:32 ORIG_HEAD
-rwxrwxrwx 1 vscode vscode  16K May 21 08:38 packed-refs
drwxrwxrwx 1 vscode vscode 4.0K May 21 08:38 refs

and where remoteUser is still set to 'root'

The other repo/container just randomly started working again (had switched back to vscode user, rebuild and restarted a few times) but it only started working again when my laptop woke up after lunch break

[chrmarti]

I'm not aware of any changes on our side that could cause this. The Docker Desktop 4.30.0 was recently released. Have you tried downgrading that? (https://docs.docker.com/desktop/release-notes/)

[ka-bu]

Yes that was one of the first things I've tried, along with downgrading the various vscode extensions.

I'll give it another try later and then update here probably tomorrow

[ka-bu]

@chrmarti worked fine on Friday and is broken today. I had disabled updates for docker and (also after multiple restarts) docker desktop still claims to be on version 4.29.0.

[ka-bu]

I've de-installed and re-installed docker desktop 4.29.0 and also rebuild the images. Git works fine again, only vscode's explorer view seems messed up (some but not all files are marked as new or modified or deleted even though they didn't change). (If I find the time, I'll try the same with 4.30.0)

[chrmarti]

VS Code uses the Git executable to check for changed files (/cc @lszomoru in case I'm missing something).

Does VS Code's Explorer view show the same status as VS Code's SCM view? Do these show the same status when you are connected locally?

[ka-bu]

Yes they do. Also git outside (windows git bash) as well as the container's git agree about the status. (Also as mentioned somewhere above, vscode's docker extension complains "Is Docker installed?", after building/while running a devcontainer)

Btw, I've heard from multiple people now that docker desktop updated, even though updates where turned off. So I suspect that it might be internal updates / side effect of the managed device that docker desktop 4.29.0 keeps breaking for me, though it seems to work fine for others (for as long as it doesn't upgrade)?

[chrmarti]

Maybe related: https://github.com/docker/for-win/issues/14083 ?

[bhaden94]

Having the same issue as well. Downgrading docker desktop to 4.29.0 did not work. We are using devcontainer from Windows and it has been working for at least 6 months in its current configuration.

On windows, all the changes are tracked properly in Git, even if changes are made within the devcontainer. However, Git is not working in the devcontainer. Sometimes it says the configuration is an issue, sometimes the .git/index file doesn't exist (when it does), and now it just says, "not a git repository" and wants me to re-initialize.

Again, all the changes in Git are being tracked on Windows, so this being a problem with mounting the .git folder seems like it has some merit.

[ka-bu]

Doesn't solve the problem for me today either. Windows Docker claims it's 4.29.0. I've not de-installed but "downgraded" via installl --ignore-version-check, and still have a broken git -- in one container. The other one ran fine and continues to run fine. Edited: Until it then broke down later.

[bhaden94]

I wanted to add that everything was working on 4.30 for at least the last couple of weeks. Only yesterday did I notice Git problems in the devcontainer, which led me to find this opened issue.

I have uninstalled 4.30 and re-installed 4.29 from the Docker Desktop version releases page. That did not fix the issue. Last night Docker force upgraded to 4.30 again. I am still seeing the problems.

At least one other teammate of mine is experiencing the same problems.

[ka-bu]

Looks like some internal update was rolled out yesterday (docker didn't just crash but was upgraded to 4.30 despite the setting that says it should not) and since then it's working fine for me. That said, something similar happened before and then the problem re-appeared. Will let you know next week, and then hopefully we can close this.

[bhaden94]

Seems to be working for me now as well. I tried upgrading to 4.31. That causes more problems. Once I re-installed docker 4.30, git works again in the devcontainer.