remote containers - podman over ssh not working

I'm trying to use vscode devcontainers with Podman over ssh (Podman has a Mac client that can connect to Linux servers). It works with Docker, but fails with Podman.

I have set up a Podman server (Fedora 33) and configured the Podman client on MacOS to connect to it via ssh:

on MacOS:

$ podman system connection ls
Name    Identity                          URI
connA*     /Users/myuser/.ssh/podman_rsa  ssh://root@myserver:22/run/podman/podman.sock
$ podman run --rm docker.io/library/hello-world
Hello from Docker!

In vscode I have a simple .devcontainer/devcontainer.json. Apart from the usual vanilla example devcontainer.json files I've only changed to use a container volume as the workspace, instead of a bind mount (as to the vscode docs on docker over ssh):

"workspaceMount": "source=vscode-workspace-myproject,target=/workspace,type=volume",

In ~/Library/Application Support/Code/User/settings.json I've set:

"docker.host": "ssh://myserver",
"remote.containers.dockerPath": "podman",

When I set remote.containers.dockerPath to docker (and make sure myserver hosts a Docker remote server instead of Podman) everything works as expected (vscode creates and connects to a devcontainer on the remote server).

When I switch this to a Podman server it doesn't work anymore. Vscode creates a container on the remote host (I can see it with podman ps), but fails to connect to it. In the startup log output this part loops (until aborted or max time reached):

[42029 ms] Start: Run: podman inspect --type container e028d16d461c355914a39445ba2dc2a4daeffbadb66a4af5290a78bf299201db
[42569 ms] Forwarding connection from port 53830 on local port 39389 to container port 39389
[42570 ms] Start: Run: podman exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=a7c9ce54-f1da-46e9-bf96-cb439c40b9cd1615392824964 e028d16d461c355914a39445ba2dc2a4daeffbadb66a4af5290a78bf299201db /root/.vscode-server/bin/f30a9b73e8ffc278e71575118b6bf568f04587c8/node -e 
[48192 ms] Start: Run: podman inspect --type container e028d16d461c355914a39445ba2dc2a4daeffbadb66a4af5290a78bf299201db
[48645 ms] Forwarding connection from port 53837 on local port 39389 to container port 39389
[48646 ms] Start: Run: podman exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=a7c9ce54-f1da-46e9-bf96-cb439c40b9cd1615392824964 e028d16d461c355914a39445ba2dc2a4daeffbadb66a4af5290a78bf299201db /root/.vscode-server/bin/f30a9b73e8ffc278e71575118b6bf568f04587c8/node -e

A podman ps shows the container:

CONTAINER ID  IMAGE                                                                COMMAND               CREATED            STATUS                PORTS   NAMES
$podman ps
e028d16d461c  localhost/vsc-devcontainer7-4f5946bffae378cdd9c0e08a38cba4f5:latest  -c echo Container...  About an hour ago  Up About an hour ago          affectionate_sutherland

and I can connect to it fine with podman exec -ti affectionate_sutherland /bin/bash, etc.

This is what it an "execsnoop"-like trace looks like:

$ sudo dtrace -s /usr/bin/newproc.d | grep -i podman
2021 Mar 10 16:49:09 13255 <13103> 64b  podman context ls --format {{json .}}
2021 Mar 10 16:49:09 13256 <13103> 64b  podman version --format {{.Server.APIVersion}}
2021 Mar 10 16:49:13 13273 <13271> 64b  podman context ls --format {{json .}}
2021 Mar 10 16:49:13 13274 <13271> 64b  podman context ls --format {{json .}}
2021 Mar 10 16:49:13 13275 <13271> 64b  podman version --format {{.Server.APIVersion}}
2021 Mar 10 16:49:13 13277 <13271> 64b  podman ps -q -a --filter label=vsch.local.folder=/Users/myuser/projects/devcontainer7 <...>
2021 Mar 10 16:49:14 13278 <13271> 64b  podman inspect --type container e028d16d461c
2021 Mar 10 16:49:14 13279 <13271> 64b  podman inspect --type container e028d16d461c355914a39445ba2dc2a4daeffbadb66a4af5290a78bf299201db
2021 Mar 10 16:49:15 13286 <13271> 64b  podman exec -i -u root -e <...>
2021 Mar 10 16:49:16 13304 <13271> 64b  podman exec -i -u root -e <...>
2021 Mar 10 16:49:16 13314 <13271> 64b  podman exec -i -u root e028d16d461c355914a39445ba2dc2a4daeffbadb66a4af5290a78bf299201db <...>
2021 Mar 10 16:49:16 13315 <13271> 64b  podman exec -i -u root -e <...>
2021 Mar 10 16:49:17 13316 <13271> 64b  podman exec -i -u root -e <...>
2021 Mar 10 16:49:18 13321 <13271> 64b  podman inspect --type container e028d16d461c355914a39445ba2dc2a4daeffbadb66a4af5290a78bf299201db
2021 Mar 10 16:49:18 13323 <13271> 64b  podman exec -i -u root -e <...>
2021 Mar 10 16:49:19 13324 <13271> 64b  podman inspect --type container e028d16d461c355914a39445ba2dc2a4daeffbadb66a4af5290a78bf299201db
2021 Mar 10 16:49:19 13325 <13271> 64b  podman exec -i -u root -e <...>
2021 Mar 10 16:49:20 13326 <13271> 64b  podman inspect --type container e028d16d461c355914a39445ba2dc2a4daeffbadb66a4af5290a78bf299201db
2021 Mar 10 16:49:21 13327 <13271> 64b  podman exec -i -u root -e <...>
[loops on until aborted]

(the <...> is a truncation by the tracing tool)

VSCode Version: 1.54.1
Local OS Version: macOS BigSur 11.2.2 x86_64 with Podman 3.0.1 client
Remote OS Version: Fedora 33 x86_64 with Podman 3.0.1 server (prepared for remoting)
Remote Extension/Connection Type: Docker (remote containers / for devcontainer)
Does this issue occur when you try this locally?: No
Does this issue occur when you try this locally and all extensions are disabled?: No

Are there any additional entries in the dev console? (Help > Toggle Developer Tools)

@chrmarti

Are there any additional entries in the dev console? (Help > Toggle Developer Tools)

below is the log output. I've also made a quick screen recording in case more context is needed: https://www.youtube.com/watch?v=w3mT7oM1P40

dev console output:

WebSocket connection to 'ws://127.0.0.1:40391/?reconnectionToken=3c41991c-4e7e-4ec9-86c8-db118b64b599&reconnection=false&skipWebSocketFrames=false' failed: Invalid frame header
create @ browserSocketFactory.ts:162
log.ts:284  INFO [remote-connection][Management   ][3c419…][reconnect] starting reconnecting loop. You can get more information with the trace log level.
log.ts:284  INFO [remote-connection][Management   ][3c419…][reconnect] resolving connection...
log.ts:284  INFO [remote-connection][Management   ][3c419…][reconnect] connecting to 127.0.0.1:40391...

[...] loops on

Dev Containers pane output:

[115050 ms] Start: Run: podman exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=ec140132-ef29-4865-92c6-a7e9b088a27a1615541333019 6cbc899e4d3af238fb31744d80bcb16144ceaa0243b3e4bd585fba84199e8325 /root/.vscode-server/bin/fd6f3bce6709b121a895d042d343d71f317d74e7/node -e 
[115939 ms] [09:30:50] [::ffff:127.0.0.1][3c41991c][ManagementConnection] The client has reconnected.
[120963 ms] Start: Run: podman inspect --type container 6cbc899e4d3af238fb31744d80bcb16144ceaa0243b3e4bd585fba84199e8325
[121713 ms] Forwarding connection from port 51655 on local port 40391 to container port 40391
[121713 ms] Start: Run: podman exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=ec140132-ef29-4865-92c6-a7e9b088a27a1615541333019 6cbc899e4d3af238fb31744d80bcb16144ceaa0243b3e4bd585fba84199e8325 /root/.vscode-server/bin/fd6f3bce6709b121a895d042d343d71f317d74e7/node -e 
[123503 ms] [09:30:58] [::ffff:127.0.0.1][3c41991c][ManagementConnection] The client has reconnected.

[loops on...]

and the dev console output with log level trace:

log.ts:272 TRACE [lifecycle] starting up (startup kind: 1)
log.ts:272 TRACE lifecycle: phase changed (value: 2)
log.ts:272 TRACE Renderer->SharedProcess#connect
log.ts:272 TRACE Skipping initializing user data in desktop
log.ts:272 TRACE Layout#layout, height: 832, width: 673
log.ts:272 TRACE workbench configuration {"_":["/Users/christian/projects/devcontainer10"],"log":"trace","crash-reporter-id":"a4a01225-3fe3-4816-a865-676498ea7e9d","logsPath":"/Users/christian/Library/Application Support/Code/logs/20210312T112058","appRoot":"/Applications/Visual Studio Code.app/Contents/Resources/app","machineId":"d4eb8e72a6d4f633ef157aaecaf8c737a04565d561e102aaa0d158e5479f6f04","nodeCachedDataDir":"/Users/christian/Library/Application Support/Code/CachedData/fd6f3bce6709b121a895d042d343d71f317d74e7","mainPid":33035,"execPath":"/Applications/Visual Studio Code.app/Contents/MacOS/Electron","userEnv":{"VSCODE_IPC_HOOK":"/Users/christian/Library/Application Support/Code/1.54.2-main.sock","VSCODE_NLS_CONFIG":"{\"locale\":\"en-gb\",\"availableLanguages\":{},\"_languagePackSupport\":true}"},"workspace":{"id":"3fba4718e97ffb00f3162acd1e0a5e21","uri":{"$mid":1,"external":"vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace","path":"/workspace","scheme":"vscode-remote","authority":"dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d"}},"remoteAuthority":"dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d","backupPath":"/Users/christian/Library/Application Support/Code/Backups/c0203006edc29a30380618b0b322e964","windowId":1,"sessionId":"window:1","logLevel":0,"colorScheme":{"dark":true,"highContrast":false},"autoDetectHighContrast":true,"maximized":true,"perfMarks":[{"name":"code/timeOrigin","startTime":1615544457365},{"name":"code/didStartMain","startTime":1615544457533},{"name":"code/mainAppReady","startTime":1615544458438},{"name":"code/willGenerateNls","startTime":1615544458439},{"name":"code/didGenerateNls","startTime":1615544458440},{"name":"code/willLoadMainBundle","startTime":1615544458468},{"name":"code/fork/willLoadCode","startTime":1615544458485},{"name":"code/registerFilesystem/file","startTime":1615544458546},{"name":"code/didLoadMainBundle","startTime":1615544458547},{"name":"code/willOpenNewWindow","startTime":1615544458769},{"name":"code/willOpenNewWindow","startTime":1615544487092}],"partsSplashPath":"/Users/christian/Library/Application Support/Code/rapid_render.json","os":{"release":"20.3.0"}}
log.ts:272 TRACE lifecycle: phase changed (value: 3)
log.ts:272 TRACE Renderer->SharedProcess#connect: connection established
log.ts:272 TRACE StartupGettingStarted: undefined
log.ts:272 TRACE Error while resolving configuration file 'vscode-userdata:/Users/christian/Library/Application%20Support/Code/User/tasks.json': Unable to read file 'vscode-userdata:/Users/christian/Library/Application Support/Code/User/tasks.json' (Error: Unable to resolve non-existing file 'vscode-userdata:/Users/christian/Library/Application Support/Code/User/tasks.json')
2log.ts:272 TRACE CommandService#executeCommand setContext
log.ts:272 TRACE MainThreadTreeViews#$registerTreeViewDataProvider targetsContainers {showCollapseAll: false, canSelectMany: false}
log.ts:272 TRACE CommandService#executeCommand setContext
log.ts:272 TRACE MainThreadTreeViews#$registerTreeViewDataProvider detailsContainers {showCollapseAll: false, canSelectMany: false}
log.ts:272 TRACE lifecycle: phase changed (value: 4)
log.ts:272 TRACE RequestService#request https://vscodeexperiments.azureedge.net/experiments/vscode-experiments.json
log.ts:272 TRACE CommandService#executeCommand remote-containers.revealLogTerminal
log.ts:272 TRACE terminalInstance#ctor (id: 1) {name: "Dev Containers", executable: undefined, args: undefined, cwd: undefined, waitOnExit: undefined, …}
log.ts:278 DEBUG Terminal process ready (shellProcessId: -1)
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][initial][127.0.0.1:36183] 1/6. invoking socketFactory.connect().
log.ts:272 TRACE [remote-connection][ExtensionHost][fa7ce…][initial][127.0.0.1:36183] 1/6. invoking socketFactory.connect().
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][initial][127.0.0.1:36183] 2/6. socketFactory.connect() was successful.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][initial][127.0.0.1:36183] 3/6. sending AuthRequest control message.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][initial][127.0.0.1:36183] 4/6. received SignRequest control message.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][initial][127.0.0.1:36183] 5/6. sending ConnectionTypeRequest control message.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][initial][127.0.0.1:36183] 6/6. handshake finished, connection is up and running after 950 ms!
log.ts:272 TRACE Error: Error: ENOENT: no such file or directory, stat '/workspace/node_modules'
log.ts:272 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json' (Error: Unable to resolve non-existing file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json')
log.ts:272 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json' (Error: Unable to resolve non-existing file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json')
log.ts:272 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json' (Error: Unable to resolve non-existing file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json')
log.ts:272 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json' (Error: Unable to resolve non-existing file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json')
log.ts:272 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json' (Error: Unable to resolve non-existing file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json')
log.ts:272 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json' (Error: Unable to resolve non-existing file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66322e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json')
log.ts:272 TRACE [remote-connection][ExtensionHost][fa7ce…][initial][127.0.0.1:36183] 2/6. socketFactory.connect() was successful.
log.ts:272 TRACE [remote-connection][ExtensionHost][fa7ce…][initial][127.0.0.1:36183] 3/6. sending AuthRequest control message.
browserSocketFactory.ts:162 WebSocket connection to 'ws://127.0.0.1:36183/?reconnectionToken=da2aabe8-25f9-488b-94a8-a0a728e11959&reconnection=false&skipWebSocketFrames=false' failed: Invalid frame header
create @ browserSocketFactory.ts:162
connect @ browserSocketFactory.ts:211
o @ remoteAgentConnection.ts:192
n @ remoteAgentConnection.ts:233
i @ remoteAgentConnection.ts:310
a @ remoteAgentConnection.ts:334
w @ remoteAgentConnection.ts:407
log.ts:284  INFO [remote-connection][Management   ][da2aa…][reconnect] starting reconnecting loop. You can get more information with the trace log level.
log.ts:284  INFO [remote-connection][Management   ][da2aa…][reconnect] resolving connection...
log.ts:272 TRACE CommandService#executeCommand setContext
log.ts:272 TRACE [remote-connection][ExtensionHost][fa7ce…][initial][127.0.0.1:36183] 4/6. received SignRequest control message.
log.ts:272 TRACE [remote-connection][ExtensionHost][fa7ce…][initial][127.0.0.1:36183] 5/6. sending ConnectionTypeRequest control message.
log.ts:272 TRACE [remote-connection][ExtensionHost][fa7ce…][initial][127.0.0.1:36183] 6/6. handshake finished, connection is up and running after 1888 ms!
log.ts:284  INFO [remote-connection][Management   ][da2aa…][reconnect] connecting to 127.0.0.1:36183...
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 1/6. invoking socketFactory.connect().
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 2/6. socketFactory.connect() was successful.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 3/6. sending AuthRequest control message.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 4/6. received SignRequest control message.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 5/6. sending ConnectionTypeRequest control message.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 6/6. handshake finished, connection is up and running after 925 ms!
log.ts:284  INFO [remote-connection][Management   ][da2aa…][reconnect] reconnected!
browserSocketFactory.ts:162 WebSocket connection to 'ws://127.0.0.1:36183/?reconnectionToken=da2aabe8-25f9-488b-94a8-a0a728e11959&reconnection=true&skipWebSocketFrames=false' failed: Invalid frame header
create @ browserSocketFactory.ts:162
connect @ browserSocketFactory.ts:211
o @ remoteAgentConnection.ts:192
n @ remoteAgentConnection.ts:233
i @ remoteAgentConnection.ts:310
a @ remoteAgentConnection.ts:334
_reconnect @ remoteAgentConnection.ts:665
_runReconnectingLoop @ remoteAgentConnection.ts:594
log.ts:284  INFO [remote-connection][Management   ][da2aa…][reconnect] starting reconnecting loop. You can get more information with the trace log level.
log.ts:284  INFO [remote-connection][Management   ][da2aa…][reconnect] resolving connection...
log.ts:272 TRACE CommandService#executeCommand setContext
log.ts:284  INFO [remote-connection][Management   ][da2aa…][reconnect] connecting to 127.0.0.1:36183...
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 1/6. invoking socketFactory.connect().
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 2/6. socketFactory.connect() was successful.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 3/6. sending AuthRequest control message.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 4/6. received SignRequest control message.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 5/6. sending ConnectionTypeRequest control message.
log.ts:272 TRACE [remote-connection][Management   ][da2aa…][reconnect][127.0.0.1:36183] 6/6. handshake finished, connection is up and running after 936 ms!
log.ts:284  INFO [remote-connection][Management   ][da2aa…][reconnect] reconnected!

[loops on...]

Maybe check if podman exec can reliably do binary data transfer in this setup:

cat <some-large-binary> | docker exec -i <container-id> cat | shasum

And compare that with shasum <some-large-binary>.

good point, doesn't look like it's reliable:

$ shasum test.dump
dec59bf51fd2e16ba07f159f7ac4b03115f7da6a  test.dump

$ while true; do cat test.dump | podman exec -ti deff7cca614f /usr/bin/cat | shasum; sleep 1; done

e1c778003878fb7689a3c11ba838c55d2c892977  -
Error: failed to create sshClient: Connection to bastion host (ssh://root@f1.tailscale.tk:22/run/podman/podman.sock) failed.: dial tcp: i/o timeout
da39a3ee5e6b4b0d3255bfef95601890afd80709  -
3420f9666aaf18dd00a610d798c20aec01790f79  -
e1c778003878fb7689a3c11ba838c55d2c892977  -
e1c778003878fb7689a3c11ba838c55d2c892977  -
e1c778003878fb7689a3c11ba838c55d2c892977  -
421f5bae95704e6f880052a19c07bb3c765dde46  -
2c6c02e52e781abd48adeb5709040d35c6fe7c91  -
bcf640cfd859d944e275e7af0c29bc4737a7d554  -
a0e5f3b5b23b7e67b0750a5711e9b8f223414437  -
e1c778003878fb7689a3c11ba838c55d2c892977  -
e1c778003878fb7689a3c11ba838c55d2c892977  -
cc7bbfda044271fada1e461f4d05331505f0923a  -
4d2234c0b4328735b1b44dca5341c862e4105741  -
e1c778003878fb7689a3c11ba838c55d2c892977  -
e1c778003878fb7689a3c11ba838c55d2c892977  -
89768deed4a916bd405db36c9748dfc609384a9b  -

sometimes (rarely) the content of the returned data is Error: container create failed (no logs from conmon): EOF, most of the time it's binary of various length. I've only seen the "bastion host timeout" once (but I think that's ok as it's a proper error).

Could you open an issue for Podman for this? We need podman exec to reliably transfer data over standard i/o for Remote-Containers to work.

I'm now working around the stdio issue(s?) and another podman/docker cli incompatibility (docker events --format {{json . }} needs to be podman --format {{.}} --format json) with a trivial wrapper for the podman command:

$ cat `which podman`
#!/bin/bash

# podman remote wrapper
if [ "${1}" == "exec" ]
    then
        ssh -T f1.tailscale.tk -o LogLevel=QUIET "podman $*"
elif [ "$*" == 'events --format {{json .}} --filter event=start' ]
    then
        /usr/local/Cellar/podman/3.0.1/bin/podman events --format {{.}} --filter event=start --format json
    else
        /usr/local/Cellar/podman/3.0.1/bin/podman $*
fi

I can now reliably transfer large binary data via stdio:

$ shasum test.dump
da5842da5eea772157607a3b53250507f56b1c1c  test.dump

$ while true; do cat test.dump | podman exec -i 134a11913b89 cat -- | shasum; sleep 1; done
da5842da5eea772157607a3b53250507f56b1c1c  -
da5842da5eea772157607a3b53250507f56b1c1c  -
da5842da5eea772157607a3b53250507f56b1c1c  -
...

and pipe commands to a container like:

$ echo $'uname\nuname -a' | podman exec -i 134a11913b89 /bin/sh
Linux
Linux testcontainer 5.10.17-200.fc33.x86_64 #1 SMP Wed Feb 17 21:21:20 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

In vscode I now get further in the process of connecting to the devcontainer, but still run into an error.

When the container doesn't exist yet I get this as part of the initial build/setup:

[...]
  vim-enhanced-2:8.2.2607-1.fc34.x86_64                                         
  vim-filesystem-2:8.2.2607-1.fc34.noarch                                       
  wget-1.21.1-2.fc34.x86_64                                                     

Complete!
--> 729539afbde
STEP 5: RUN mkdir /workspace
--> 5be7c07be87
STEP 6: WORKDIR /workspace
--> 4a0eee0f5b9
STEP 7: CMD /bin/bash
STEP 8: COMMIT vsc-devcontainer10-a57cf9f61fcb18f79530ad9aa379d3a9
--> 03a18d659ab
03a18d659ab913d6b2cedd46331b9460beb78071e15543f1f53b0ccb81925d5c
[182605 ms] Start: Run: podman events --format {{json .}} --filter event=start
[182608 ms] Start: Starting container
[182608 ms] Start: Run: podman run --sig-proxy=false -a STDOUT -a STDERR --mount source=vscode-workspace-myproject,target=/workspace,type=volume --mount type=volume,src=vscode,dst=/vscode -l vsch.local.folder=/Users/christian/projects/devcontainer10 -l vsch.quality=stable -l vsch.remote.devPort=0 --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --entrypoint /bin/sh vsc-devcontainer10-a57cf9f61fcb18f79530ad9aa379d3a9 -c echo Container started ; trap "exit 0" 15; while sleep 1 & wait $!; do :; done

[184428 ms] Start: Run: podman ps -q -a --filter label=vsch.local.folder=/Users/christian/projects/devcontainer10 --filter label=vsch.quality=stable
[184459 ms] Start: Run: podman inspect --type container 534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0
[185088 ms] Start: Run: podman inspect --type container 534d3de92297
[186177 ms] [object Object]

the output of podman inspect --type container 534d3de92297:

podman inspect --type container 534d3de92297[
    {
        "Id": "534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0",
        "Created": "2021-03-19T13:38:50.425311443+01:00",
        "Path": "/bin/sh",
        "Args": [
            "-c",
            "echo",
            "Container",
            "started",
            ";",
            "trap",
            "\"exit",
            "0\"",
            "15;",
            "while",
            "sleep",
            "1",
            "\u0026",
            "wait",
            "$!;",
            "do",
            ":;",
            "done"
        ],
        "State": {
            "OciVersion": "1.0.2-dev",
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-03-19T13:38:50.988809185+01:00",
            "FinishedAt": "2021-03-19T13:38:50.996429575+01:00",
            "Healthcheck": {
                "Status": "",
                "FailingStreak": 0,
                "Log": null
            }
        },
        "Image": "03a18d659ab913d6b2cedd46331b9460beb78071e15543f1f53b0ccb81925d5c",
        "ImageName": "localhost/vsc-devcontainer10-a57cf9f61fcb18f79530ad9aa379d3a9:latest",
        "Rootfs": "",
        "Pod": "",
        "ResolvConfPath": "/var/run/containers/storage/overlay-containers/534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0/userdata/resolv.conf",
        "HostnamePath": "/var/run/containers/storage/overlay-containers/534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0/userdata/hostname",
        "HostsPath": "/var/run/containers/storage/overlay-containers/534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0/userdata/hosts",
        "StaticDir": "/var/lib/containers/storage/overlay-containers/534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0/userdata",
        "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0/userdata/config.json",
        "OCIRuntime": "crun",
        "ConmonPidFile": "/var/run/containers/storage/overlay-containers/534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0/userdata/conmon.pid",
        "Name": "infallible_gauss",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "system_u:object_r:container_file_t:s0:c88,c834",
        "ProcessLabel": "system_u:system_r:container_t:s0:c88,c834",
        "AppArmorProfile": "",
        "EffectiveCaps": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_NET_BIND_SERVICE",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT",
            "CAP_SYS_PTRACE"
        ],
        "BoundingCaps": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_NET_BIND_SERVICE",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT",
            "CAP_SYS_PTRACE"
        ],
        "ExecIDs": [],
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/efe5e1c6bc9e5ca3e71223dec1da98cba92df89f0ad8f63ed1db94e6c2c54683/diff:/var/lib/containers/storage/overlay/6f3fd878cb408ef602256e578c4646f8ec2beba65f5c3bf80a3c1a388babffc8/diff:/var/lib/containers/storage/overlay/4beb838099897a72f66c5cc8e7ffcf53c3be2c06d7f4fb6d1b74b160d760e2d1/diff:/var/lib/containers/storage/overlay/ae1e88b39cdea077f993a73f7c7e7e7bde0b6623b846533e73a8e62ddcce8865/diff:/var/lib/containers/storage/overlay/e542f72641415cf30c1af869cb49ce16a89e46db071bead7812ad15fbc3cb068/diff",
                "UpperDir": "/var/lib/containers/storage/overlay/b5fdbfa79ea6c2d39beec0edab25734700328a0bab6622093f9c7b071822cec4/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/b5fdbfa79ea6c2d39beec0edab25734700328a0bab6622093f9c7b071822cec4/work"
            }
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "vscode-workspace-myproject",
                "Source": "/var/lib/containers/storage/volumes/vscode-workspace-myproject/_data",
                "Destination": "/workspace",
                "Driver": "local",
                "Mode": "",
                "Options": [
                    "nosuid",
                    "nodev",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "vscode",
                "Source": "/var/lib/containers/storage/volumes/vscode/_data",
                "Destination": "/vscode",
                "Driver": "local",
                "Mode": "",
                "Options": [
                    "nosuid",
                    "nodev",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Dependencies": [],
        "NetworkSettings": {
            "EndpointID": "",
            "Gateway": "",
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "",
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "",
            "Networks": {
                "podman": {
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "NetworkID": "podman",
                    "DriverOpts": null,
                    "IPAMConfig": null,
                    "Links": null
                }
            }
        },
        "ExitCommand": [
            "/usr/bin/podman",
            "--root",
            "/var/lib/containers/storage",
            "--runroot",
            "/var/run/containers/storage",
            "--log-level",
            "info",
            "--cgroup-manager",
            "systemd",
            "--tmpdir",
            "/var/run/libpod",
            "--runtime",
            "crun",
            "--storage-driver",
            "overlay",
            "--storage-opt",
            "overlay.mountopt=nodev,metacopy=on",
            "--events-backend",
            "journald",
            "container",
            "cleanup",
            "534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0"
        ],
        "Namespace": "",
        "IsInfra": false,
        "Config": {
            "Hostname": "534d3de92297",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "container=oci",
                "DISTTAG=f34container",
                "FGC=f34",
                "HOME=/root",
                "HOSTNAME=534d3de92297"
            ],
            "Cmd": [
                "-c",
                "echo",
                "Container",
                "started",
                ";",
                "trap",
                "\"exit",
                "0\"",
                "15;",
                "while",
                "sleep",
                "1",
                "\u0026",
                "wait",
                "$!;",
                "do",
                ":;",
                "done"
            ],
            "Image": "localhost/vsc-devcontainer10-a57cf9f61fcb18f79530ad9aa379d3a9:latest",
            "Volumes": null,
            "WorkingDir": "/workspace",
            "Entrypoint": "/bin/sh",
            "OnBuild": null,
            "Labels": {
                "io.buildah.version": "1.19.4",
                "license": "MIT",
                "name": "fedora",
                "vendor": "Fedora Project",
                "version": "34",
                "vsch.local.folder": "/Users/christian/projects/devcontainer10",
                "vsch.quality": "stable",
                "vsch.remote.devPort": "0"
            },
            "Annotations": {
                "io.container.manager": "libpod",
                "io.kubernetes.cri-o.Created": "2021-03-19T13:38:50.425311443+01:00",
                "io.kubernetes.cri-o.TTY": "false",
                "io.podman.annotations.autoremove": "FALSE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.privileged": "FALSE",
                "io.podman.annotations.publish-all": "FALSE",
                "io.podman.annotations.seccomp": "unconfined",
                "org.opencontainers.image.stopSignal": "15"
            },
            "StopSignal": 15,
            "CreateCommand": [
                "/usr/local/Cellar/podman/3.0.1/bin/podman",
                "run",
                "--sig-proxy=false",
                "-a",
                "STDOUT",
                "-a",
                "STDERR",
                "--mount",
                "source=vscode-workspace-myproject,target=/workspace,type=volume",
                "--mount",
                "type=volume,src=vscode,dst=/vscode",
                "-l",
                "vsch.local.folder=/Users/christian/projects/devcontainer10",
                "-l",
                "vsch.quality=stable",
                "-l",
                "vsch.remote.devPort=0",
                "--cap-add=SYS_PTRACE",
                "--security-opt",
                "seccomp=unconfined",
                "--entrypoint",
                "/bin/sh",
                "vsc-devcontainer10-a57cf9f61fcb18f79530ad9aa379d3a9",
                "-c",
                "echo",
                "Container",
                "started",
                ";",
                "trap",
                "\"exit",
                "0\"",
                "15;",
                "while",
                "sleep",
                "1",
                "\u0026",
                "wait",
                "$!;",
                "do",
                ":;",
                "done"
            ],
            "Umask": "0022"
        },
        "HostConfig": {
            "Binds": [
                "vscode-workspace-myproject:/workspace:rw,rprivate,nosuid,nodev,rbind",
                "vscode:/vscode:rw,rprivate,nosuid,nodev,rbind"
            ],
            "CgroupManager": "systemd",
            "CgroupMode": "private",
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "k8s-file",
                "Config": null,
                "Path": "/var/lib/containers/storage/overlay-containers/534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0/userdata/ctr.log",
                "Tag": "",
                "Size": "0B"
            },
            "NetworkMode": "bridge",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [
                "CAP_SYS_PTRACE"
            ],
            "CapDrop": [
                "CAP_MKNOD",
                "CAP_NET_RAW",
                "CAP_AUDIT_WRITE"
            ],
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": [],
            "IpcMode": "private",
            "Cgroup": "",
            "Cgroups": "default",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "private",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "seccomp=unconfined"
            ],
            "Tmpfs": {},
            "UTSMode": "private",
            "UsernsMode": "",
            "ShmSize": 65536000,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": 0,
            "OomKillDisable": false,
            "PidsLimit": 2048,
            "Ulimits": [
                {
                    "Name": "RLIMIT_NOFILE",
                    "Soft": 1048576,
                    "Hard": 1048576
                },
                {
                    "Name": "RLIMIT_NPROC",
                    "Soft": 4194304,
                    "Hard": 4194304
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "CgroupConf": null
        }
    }
]

from the dev console log:

/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE [lifecycle] starting up (startup kind: 1)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE lifecycle: phase changed (value: 2)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Renderer->SharedProcess#connect
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Skipping initializing user data in desktop
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Layout#layout, height: 832, width: 1440
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE workbench configuration {"_":["/Users/christian/projects/devcontainer10"],"log":"trace","crash-reporter-id":"a4a01225-3fe3-4816-a865-676498ea7e9d","logsPath":"/Users/christian/Library/Application Support/Code/logs/20210319T133531","appRoot":"/Applications/Visual Studio Code.app/Contents/Resources/app","machineId":"d4eb8e72a6d4f633ef157aaecaf8c737a04565d561e102aaa0d158e5479f6f04","nodeCachedDataDir":"/Users/christian/Library/Application Support/Code/CachedData/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8","mainPid":65389,"execPath":"/Applications/Visual Studio Code.app/Contents/MacOS/Electron","userEnv":{"VSCODE_IPC_HOOK":"/Users/christian/Library/Application Support/Code/1.54.3-main.sock","VSCODE_NLS_CONFIG":"{\"locale\":\"en-gb\",\"availableLanguages\":{},\"_languagePackSupport\":true}"},"workspace":{"id":"b12680432adf46e71f0c5950172de240","uri":{"$mid":1,"external":"vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace","path":"/workspace","scheme":"vscode-remote","authority":"dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d"}},"remoteAuthority":"dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d","backupPath":"/Users/christian/Library/Application Support/Code/Backups/2261b3e95959bfd71ff60a1c0063e713","windowId":1,"sessionId":"window:1","logLevel":0,"colorScheme":{"dark":true,"highContrast":false},"autoDetectHighContrast":true,"maximized":true,"perfMarks":[{"name":"code/timeOrigin","startTime":1616157330020},{"name":"code/didStartMain","startTime":1616157330181},{"name":"code/mainAppReady","startTime":1616157331278},{"name":"code/willGenerateNls","startTime":1616157331279},{"name":"code/didGenerateNls","startTime":1616157331280},{"name":"code/willLoadMainBundle","startTime":1616157331299},{"name":"code/fork/willLoadCode","startTime":1616157331316},{"name":"code/registerFilesystem/file","startTime":1616157331375},{"name":"code/didLoadMainBundle","startTime":1616157331376},{"name":"code/willOpenNewWindow","startTime":1616157331579},{"name":"code/willOpenNewWindow","startTime":1616157344621}],"partsSplashPath":"/Users/christian/Library/Application Support/Code/rapid_render.json","os":{"release":"20.3.0"}}
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE lifecycle: phase changed (value: 3)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Renderer->SharedProcess#connect: connection established
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Error while resolving configuration file 'vscode-userdata:/Users/christian/Library/Application%20Support/Code/User/tasks.json': Unable to read file 'vscode-userdata:/Users/christian/Library/Application Support/Code/User/tasks.json' (Error: Unable to resolve non-existing file 'vscode-userdata:/Users/christian/Library/Application Support/Code/User/tasks.json')
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE StartupGettingStarted: false
2/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE CommandService#executeCommand setContext
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE MainThreadTreeViews#$registerTreeViewDataProvider targetsContainers Object
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE CommandService#executeCommand setContext
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE MainThreadTreeViews#$registerTreeViewDataProvider detailsContainers Object
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE lifecycle: phase changed (value: 4)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE RequestService#request https://vscodeexperiments.azureedge.net/experiments/vscode-experiments.json
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE CommandService#executeCommand remote-containers.revealLogTerminal
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE terminalInstance#ctor (id: 1) Object
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 DEBUG Terminal process ready (shellProcessId: -1)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE DialogService#show An error occurred setting up the container.
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2766 Error
    at nr.resolveAuthority (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1719)
    at async x._scanAndHandleExtensions (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2766)
    at async x._initialize (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1857)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2766 Error handled: Not showing a notification for the error
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1719 Error received from starting extension host
(anonymous) @ /Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1719
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1719 Error
    at nr.resolveAuthority (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1719)
    at async x._scanAndHandleExtensions (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2766)
    at async x._initialize (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1857)
(anonymous) @ /Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1719
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626   ERR [remote-connection] An error occurred in the very first connect attempt, it will be treated as a permanent error! Error:
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626   ERR Error
    at nr.resolveAuthority (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1719)
    at async x._scanAndHandleExtensions (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2766)
    at async x._initialize (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1857)
2/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:vscode-userdata
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:vscode-remote
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Settings Sync: Initializing accounts
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Settings Sync: Getting accounts for microsoft
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onAuthenticationRequest:microsoft
2/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:file
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:vscode-userdata
3/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:vscode-remote
10/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:file
2/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:vscode-remote
5/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:file
6/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:vscode-userdata
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:vscode-remote
5/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:file
2/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onCommand:setContext
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:file
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onCommand:setContext
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:file
5/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:vscode-remote
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onCommand:remote-containers.revealLogTerminal
10/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onFileSystem:file
2/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Error
    at nr.resolveAuthority (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1719)
    at async x._scanAndHandleExtensions (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2766)
    at async x._initialize (/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1857)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626  WARN Ignoring the error while validating workspace folder vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace - An unknown error occurred. Please consult the log for more details.
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626   ERR An unknown error occurred. Please consult the log for more details.: Error
    at nr.resolveAuthority (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1719:7977)
    at async x._scanAndHandleExtensions (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2766:6446)
    at async x._initialize (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1857:15232)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json' (Error)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626   ERR Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json' (Error): Error: Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json' (Error)
    at s.doReadAsFileStream (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:29459)
    at async s.readFile (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:28581)
    at async file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:5717
    at async Promise.all (index 0)
    at async Promise.all (index 0)
    at async b.loadConfiguration (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:6373)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json' (Error)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626   ERR Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json' (Error): Error: Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json' (Error)
    at s.doReadAsFileStream (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:29459)
    at async s.readFile (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:28581)
    at async file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:5717
    at async Promise.all (index 0)
    at async Promise.all (index 1)
    at async b.loadConfiguration (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:6373)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json' (Error)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626   ERR Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json' (Error): Error: Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json' (Error)
    at s.doReadAsFileStream (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:29459)
    at async s.readFile (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:28581)
    at async file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:5717
    at async Promise.all (index 1)
    at async Promise.all (index 1)
    at async b.loadConfiguration (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:6373)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json' (Error)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626   ERR Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json' (Error): Error: Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/settings.json' (Error)
    at s.doReadAsFileStream (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:29459)
    at async s.readFile (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:28581)
    at async file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:5717
    at async Promise.all (index 0)
    at async Promise.all (index 0)
    at async b.loadConfiguration (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:6373)
    at async Promise.all (index 0)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json' (Error)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626   ERR Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json' (Error): Error: Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/tasks.json' (Error)
    at s.doReadAsFileStream (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:29459)
    at async s.readFile (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:28581)
    at async file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:5717
    at async Promise.all (index 0)
    at async Promise.all (index 1)
    at async b.loadConfiguration (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:6373)
    at async Promise.all (index 0)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Error while resolving configuration file 'vscode-remote://dev-container%2B7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json': Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json' (Error)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626   ERR Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json' (Error): Error: Unable to read file 'vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f63687269737469616e2f70726f6a656374732f646576636f6e7461696e65723130222c2273657474696e6773223a7b22686f7374223a227373683a2f2f66312e7461696c7363616c652e746b227d7d/workspace/.vscode/launch.json' (Error)
    at s.doReadAsFileStream (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:29459)
    at async s.readFile (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:28581)
    at async file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:5717
    at async Promise.all (index 1)
    at async Promise.all (index 1)
    at async b.loadConfiguration (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1233:6373)
    at async Promise.all (index 0)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626   ERR An unknown error occurred. Please consult the log for more details.: Error
    at nr.resolveAuthority (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1719:7977)
    at async x._scanAndHandleExtensions (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2766:6446)
    at async x._initialize (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1857:15232)
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Settings Sync: Updated accounts for microsoft
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Settings Sync: Getting accounts for github
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE onWillActivateByEvent:  onAuthenticationRequest:github
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Settings Sync: Updated accounts for github
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Settings Sync: Account status changed from uninitialized to unavailable
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE Settings Sync: Accounts are initialized
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE KeybindingService#dispatch ⌘C workbench.action.terminal.copySelection
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE CommandService#executeCommand workbench.action.terminal.copySelection
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE KeybindingService#dispatch ⌘C workbench.action.terminal.copySelection
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE CommandService#executeCommand workbench.action.terminal.copySelection
/Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626 TRACE CommandService#executeCommand workbench.action.toggleDevTools
log.ts:272 TRACE Layout#layout, height: 832, width: 673
log.ts:296   ERR An unknown error occurred. Please consult the log for more details.: Error
    at nr.resolveAuthority (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1719:7977)
    at async x._scanAndHandleExtensions (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2766:6446)
    at async x._initialize (file:///Applications/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:1857:15232)

And when I then close/reopen vscode and try to reconnect to the container at this stage:

[5 ms] Remote-Containers 0.163.2 in VS Code 1.54.3 (2b9aebd5354a3629c3aba0a5f5df49f43d6689f8).
[114 ms] Start: Resolving Remote
[116 ms] Setting up container for folder or workspace: /Users/christian/projects/devcontainer10
[116 ms] Host: ssh://f1.tailscale.tk
[123 ms] Start: Check Docker is running
[124 ms] Start: Run: podman version --format {{.Server.APIVersion}}
[782 ms] Server API version: 3.0.0
[810 ms] Start: Run: podman ps -q -a --filter label=vsch.local.folder=/Users/christian/projects/devcontainer10 --filter label=vsch.quality=stable
[1362 ms] Start: Run: podman inspect --type container 534d3de92297
[1977 ms] Start: Starting container
[1977 ms] Start: Run: podman start 534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0
[2931 ms] Start: Run: podman ps -q -a --filter label=vsch.local.folder=/Users/christian/projects/devcontainer10 --filter label=vsch.quality=stable
[3681 ms] Start: Run: podman inspect --type container 534d3de92297
[4330 ms] Start: Inspecting container
[4330 ms] Start: Run: podman inspect --type container 534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0
[4950 ms] Start: Run: podman exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=7659d927-d395-4ada-8d2d-0c5d933d8a4e1616158208039 534d3de92297f4781431e05b74dd6cc08a2f16419fcb10a9d59a93a21c303ee0 /bin/sh
[4954 ms] Start: Run in container: uname -m
[5598 ms] Start: Run in container: cat /etc/passwd
[5598 ms] Stdin closed!
[5602 ms] Shell server terminated (code: 255, signal: null)

Error: can only create exec sessions on running containers: container state improper

Any ideas?

"podman $*" appears to drop the split into arguments. See the "Args" property of the output from inspect, that should be only 2 arguments -c and the short shell script.

Actually, this is podman run that runs this shell script. That you can fix by using "$@" instead of $*, see https://stackoverflow.com/questions/3811345/how-to-pass-all-arguments-passed-to-my-bash-script-to-a-function-of-mine. Not sure if that works for the exec case too, given that you already quote to have a single parameter.

thanks! ok, my wrapper now looks like this

cat `which podman`
#!/bin/bash

# podman remote wrapper
if [ "${1}" == "exec" ]
    then
        ssh -T f1.tailscale.tk -o LogLevel=QUIET "podman $@"
elif [ "$*" == 'events --format {{json .}} --filter event=start' ]
    then
        /usr/local/Cellar/podman/3.0.1/bin/podman events --log-level=error --format {{.}} --filter event=start --format json
    else
        /usr/local/Cellar/podman/3.0.1/bin/podman --log-level=error "$@"
fi

and I'm getting further, but running into another error:

  vim-filesystem-2:8.2.2607-1.fc34.noarch                                       
  wget-1.21.1-2.fc34.x86_64                                                     

Complete!
--> 4ea75b5062e
STEP 5: RUN mkdir /workspace
--> bc8c662a882
STEP 6: WORKDIR /workspace
--> a999c414744
STEP 7: CMD /bin/bash
STEP 8: COMMIT vsc-devcontainer10-a57cf9f61fcb18f79530ad9aa379d3a9
--> 7c56ae0a8da
7c56ae0a8da8283fa79290f0dcca869e2cdb4cff82a81f08524bf13838c0d212
[275425 ms] Start: Run: podman events --format {{json .}} --filter event=start
[275426 ms] Start: Starting container
[275427 ms] Start: Run: podman run --sig-proxy=false -a STDOUT -a STDERR --mount source=vscode-workspace-myproject,target=/workspace,type=volume --mount type=volume,src=vscode,dst=/vscode -l vsch.local.folder=/Users/christian/projects/devcontainer10 -l vsch.quality=stable -l vsch.remote.devPort=0 --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --entrypoint /bin/sh vsc-devcontainer10-a57cf9f61fcb18f79530ad9aa379d3a9 -c echo Container started ; trap "exit 0" 15; while sleep 1 & wait $!; do :; done
Container started
[277391 ms] Start: Run: podman inspect --type container bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90
[278111 ms] Start: Run: podman ps -q -a --filter label=vsch.local.folder=/Users/christian/projects/devcontainer10 --filter label=vsch.quality=stable
[278759 ms] Start: Run: podman inspect --type container bd34f2e6887f
[279571 ms] Start: Inspecting container
[279571 ms] Start: Run: podman inspect --type container bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90
[280213 ms] Start: Run: podman exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=69b124f5-f5e8-4ca2-813e-9122f5c27a131616162042763 bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90 /bin/sh
[280215 ms] Start: Run in container: uname -m
[281046 ms] x86_64
[281046 ms] 
[281046 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[281097 ms] NAME=Fedora
VERSION="34 (Container Image Prerelease)"
ID=fedora
VERSION_ID=34
VERSION_CODENAME=""
PLATFORM_ID="platform:f34"
PRETTY_NAME="Fedora 34 (Container Image Prerelease)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:34"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/34/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=34
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=34
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container
[281098 ms] 
[281098 ms] Start: Run in container: cat /etc/passwd
[281130 ms] Start: Updating configuration state
[281135 ms] Start: Setup shutdown monitor
[281137 ms] Forking shutdown monitor: /Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/shutdown/shutdownMonitorProcess /var/folders/1b/w55p1_bd6jqcf2hb7qn30q2w0000gn/T/vscode-remote-containers-bb116f09854704c832b95eb14d530f317e395297.sock singleContainer Info /Users/christian/Library/Application Support/Code/logs/20210319T145349/exthost1/ms-vscode-remote.remote-containers 1616162044159
[281141 ms] Start: Run in container: test -d /root/.vscode-server
[281174 ms] 
[281174 ms] 
[281174 ms] Exit code 1
[281174 ms] Start: Run in container: test -d /root/.vscode-remote
[281210 ms] 
[281210 ms] 
[281210 ms] Exit code 1
[281210 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[281243 ms] 
[281243 ms] 
[281243 ms] Start: Run in container: mkdir -p '/root/.vscode-server/data/Machine' && cat >'/root/.vscode-server/data/Machine/settings.json' <<'settingsJSON'
[281281 ms] 
[281281 ms] 
[281282 ms] Start: Run in container: test -d /root/.vscode-server/bin/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8
[281320 ms] 
[281320 ms] 
[281320 ms] Exit code 1
[281320 ms] Start: Run in container: test -d /vscode/vscode-server/bin/x64/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8
[281350 ms] 
[281350 ms] 
[281350 ms] Start: Run in container: mkdir -p '/root/.vscode-server/bin' && ln -s '/vscode/vscode-server/bin/x64/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8' '/root/.vscode-server/bin/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8'
[281384 ms] 
[281384 ms] 
[281384 ms] Start: Run in container: touch '/vscode/vscode-server/bin/x64/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8'
[281384 ms] Start: Launching Remote-Containers helper.
[281385 ms] Start: Run: gpgconf --list-dir agent-extra-socket
[281414 ms] 
[281414 ms] 
[281414 ms] Start: Run in container: gpgconf --list-dir agent-socket
[281452 ms] /root/.gnupg/S.gpg-agent
[281452 ms] 
[281452 ms] Start: Run in container: gpgconf --list-dir homedir
[281485 ms] /root/.gnupg
[281486 ms] 
[281486 ms] Start: Run in container: ls '/root/.gnupg/private-keys-v1.d' 2>/dev/null
[281523 ms] 
[281524 ms] 
[281524 ms] Exit code 2
[281524 ms] Start: Run in container: mkdir -p -m 700 '/root/.gnupg'
[281561 ms] 
[281561 ms] 
[281561 ms] Start: Run in container: # Test for /root/.ssh/known_hosts and ssh
[281562 ms] Start: Run: gpgconf --list-dir homedir
[281595 ms] 
[281596 ms] 
[281596 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.js' >/tmp/vscode-remote-containers-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.js
[281637 ms] 
[281638 ms] 
[281638 ms] Start: Run in container: gpgconf --list-dir homedir
[281670 ms] /root/.gnupg
[281671 ms] 
[281671 ms] Start: Run in container: # Copy /Users/christian/.ssh/known_hosts to /root/.ssh/known_hosts
[281738 ms] 
[281738 ms] 
[281739 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.js' >/tmp/vscode-remote-containers-server-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.js
[281928 ms] 
[281928 ms] 
[281928 ms] Start: Run in container: # Test for /root/.gnupg/pubring.kbx and gpg
[281929 ms] Start: Run: podman exec -i -u root -e REMOTE_CONTAINERS_SOCKETS=["/tmp/vscode-ssh-auth-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.sock","/root/.gnupg/S.gpg-agent"] -e REMOTE_CONTAINERS_IPC=/tmp/vscode-remote-containers-ipc-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.sock bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90 /root/.vscode-server/bin/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8/node /tmp/vscode-remote-containers-server-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.js
[281964 ms] 
[281964 ms] 
[281965 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --system credential.helper '!f() { /root/.vscode-server/bin/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8/node /tmp/vscode-remote-containers-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.js $*; }; f' || true
[282005 ms] 
[282005 ms] 
[282005 ms] Start: Run in container: # Copy /Users/christian/.gnupg/pubring.kbx to /root/.gnupg/pubring.kbx
[282005 ms] Start: Preparing Extensions
[283267 ms] Remote-Containers server: undefined:1
[/tmp/vscode-ssh-auth-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.sock,/root/.gnupg/S.gpg-agent]
 ^

SyntaxError: Unexpected token / in JSON at position 1
    at JSON.parse (<anonymous>)
    at Object.<anonymous> (/tmp/vscode-remote-containers-server-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.js:6:6688)
    at Module._compile (internal/modules/cjs/loader.js:1137:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
    at Module.load (internal/modules/cjs/loader.js:985:32)
    at Function.Module._load (internal/modules/cjs/loader.js:878:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47
[283392 ms] Remote-Containers server terminated (code: 1, signal: null).
[283837 ms] 
[283837 ms] 
[283837 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.installExtensionsMarker' ; } 2> /dev/null
[283873 ms] 
[283873 ms] 
[283873 ms] Start: Run in container: # Test for /root/.gnupg/trustdb.gpg and gpg
[283875 ms] Extensions cache, install extensions: golang.Go, redhat.vscode-yaml, ms-python.python, streetsidesoftware.code-spell-checker, esbenp.prettier-vscode, ecmel.vscode-html-css, oderwat.indent-rainbow, coenraads.bracket-pair-colorizer, ms-vscode.cpptools, mrmlnc.vscode-duplicate, aws-scripting-guy.cform, ms-azuretools.vscode-docker, dbaeumer.vscode-eslint
[283908 ms] 
[283909 ms] 
[283909 ms] Start: Run in container: test -d /root/.vscode-server/extensionsCache && ls /root/.vscode-server/extensionsCache || true
[283939 ms] 
[283940 ms] 
[283940 ms] Start: Run in container: # Copy /Users/christian/.gnupg/trustdb.gpg to /root/.gnupg/trustdb.gpg
[283999 ms] 
[283999 ms] 
[283999 ms] Start: Run in container: test -d /vscode/vscode-server/extensionsCache && ls /vscode/vscode-server/extensionsCache || true
[284000 ms] Start: Run: gpg-connect-agent updatestartuptty /bye
[284037 ms] aws-scripting-guy.cform-0.0.24
coenraads.bracket-pair-colorizer-1.0.61
dbaeumer.vscode-eslint-2.1.14
dbaeumer.vscode-eslint-2.1.16
dbaeumer.vscode-eslint-2.1.17
ecmel.vscode-html-css-1.10.2
esbenp.prettier-vscode-5.9.2
golang.go-0.22.1
golang.go-0.23.0
golang.go-0.23.2
mrmlnc.vscode-duplicate-1.2.1
ms-azuretools.vscode-docker-1.10.0
ms-python.python-2021.2.582707922
ms-python.python-2021.2.636928669
ms-toolsai.jupyter-2021.2.603412351
ms-toolsai.jupyter-2021.3.619093157
ms-vscode.cpptools-1.2.2
oderwat.indent-rainbow-7.5.0
redhat.vscode-yaml-0.15.0
redhat.vscode-yaml-0.16.0
streetsidesoftware.code-spell-checker-1.10.2
[284038 ms] 
[284038 ms] Extensions cache, link in container: aws-scripting-guy.cform-0.0.24, coenraads.bracket-pair-colorizer-1.0.61, dbaeumer.vscode-eslint-2.1.14, dbaeumer.vscode-eslint-2.1.16, dbaeumer.vscode-eslint-2.1.17, ecmel.vscode-html-css-1.10.2, esbenp.prettier-vscode-5.9.2, golang.go-0.22.1, golang.go-0.23.0, golang.go-0.23.2, mrmlnc.vscode-duplicate-1.2.1, ms-azuretools.vscode-docker-1.10.0, ms-python.python-2021.2.582707922, ms-python.python-2021.2.636928669, ms-vscode.cpptools-1.2.2, oderwat.indent-rainbow-7.5.0, redhat.vscode-yaml-0.15.0, redhat.vscode-yaml-0.16.0, streetsidesoftware.code-spell-checker-1.10.2
[284038 ms] Start: Run in container: mkdir -p '/root/.vscode-server/extensionsCache' && ln -s '/vscode/vscode-server/extensionsCache'/* '/root/.vscode-server/extensionsCache' || true
[284073 ms] 
[284073 ms] 
[284074 ms] Start: Run in container: cd /vscode/vscode-server/extensionsCache && touch 'aws-scripting-guy.cform-0.0.24' 'coenraads.bracket-pair-colorizer-1.0.61' 'dbaeumer.vscode-eslint-2.1.14' 'dbaeumer.vscode-eslint-2.1.16' 'dbaeumer.vscode-eslint-2.1.17' 'ecmel.vscode-html-css-1.10.2' 'esbenp.prettier-vscode-5.9.2' 'golang.go-0.22.1' 'golang.go-0.23.0' 'golang.go-0.23.2' 'mrmlnc.vscode-duplicate-1.2.1' 'ms-azuretools.vscode-docker-1.10.0' 'ms-python.python-2021.2.582707922' 'ms-python.python-2021.2.636928669' 'ms-vscode.cpptools-1.2.2' 'oderwat.indent-rainbow-7.5.0' 'redhat.vscode-yaml-0.15.0' 'redhat.vscode-yaml-0.16.0' 'streetsidesoftware.code-spell-checker-1.10.2'
[284109 ms] 
[284109 ms] 
[284109 ms] Start: Run in container: for pid in `cd /proc && ls -d [0-9]*`; do { echo $pid ; readlink /proc/$pid/cwd ; readlink /proc/$pid/ns/mnt ; cat /proc/$pid/stat | tr "
[284214 ms] Start: Starting VS Code Server
[284214 ms] Start: Run in container: /root/.vscode-server/bin/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8/server.sh --force-disable-user-env --port 0 --extensions-download-dir /root/.vscode-server/extensionsCache --install-extension golang.Go --install-extension redhat.vscode-yaml --install-extension ms-python.python --install-extension streetsidesoftware.code-spell-checker --install-extension esbenp.prettier-vscode --install-extension ecmel.vscode-html-css --install-extension oderwat.indent-rainbow --install-extension coenraads.bracket-pair-colorizer --install-extension ms-vscode.cpptools --install-extension mrmlnc.vscode-duplicate --install-extension aws-scripting-guy.cform --install-extension ms-azuretools.vscode-docker --install-extension dbaeumer.vscode-eslint --start-server
[284219 ms] Error: stream is closed
        at /Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:12:4122
        at i (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:12:2234)
        at EventEmitter.t.<computed> [as exec] (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:12:1959)
        at u (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:88:1014)
        at new Promise (<anonymous>)
        at /Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:88:992
        at Tr (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:129:1953)
        at kj (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:129:384)
        at processTicksAndRejections (internal/process/task_queues.js:97:5)
        at async zn (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:102:6698)
        at async Ag (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:186:9372)
        at async PP (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:186:6756)
        at async lV (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:167:7710)

some more details:

$ podman inspect bd34f2e6887f
[
    {
        "Id": "bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90",
        "Created": "2021-03-19T14:58:40.933062845+01:00",
        "Path": "/bin/sh",
        "Args": [
            "-c",
            "echo Container started ; trap \"exit 0\" 15; while sleep 1 \u0026 wait $!; do :; done"
        ],
        "State": {
            "OciVersion": "1.0.2-dev",
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 24776,
            "ConmonPid": 24773,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-03-19T14:58:41.470332635+01:00",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Healthcheck": {
                "Status": "",
                "FailingStreak": 0,
                "Log": null
            }
        },
        "Image": "7c56ae0a8da8283fa79290f0dcca869e2cdb4cff82a81f08524bf13838c0d212",
        "ImageName": "localhost/vsc-devcontainer10-a57cf9f61fcb18f79530ad9aa379d3a9:latest",
        "Rootfs": "",
        "Pod": "",
        "ResolvConfPath": "/var/run/containers/storage/overlay-containers/bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90/userdata/resolv.conf",
        "HostnamePath": "/var/run/containers/storage/overlay-containers/bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90/userdata/hostname",
        "HostsPath": "/var/run/containers/storage/overlay-containers/bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90/userdata/hosts",
        "StaticDir": "/var/lib/containers/storage/overlay-containers/bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90/userdata",
        "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90/userdata/config.json",
        "OCIRuntime": "crun",
        "ConmonPidFile": "/var/run/containers/storage/overlay-containers/bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90/userdata/conmon.pid",
        "Name": "affectionate_ptolemy",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "system_u:object_r:container_file_t:s0:c611,c741",
        "ProcessLabel": "system_u:system_r:container_t:s0:c611,c741",
        "AppArmorProfile": "",
        "EffectiveCaps": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_NET_BIND_SERVICE",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT",
            "CAP_SYS_PTRACE"
        ],
        "BoundingCaps": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_NET_BIND_SERVICE",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYS_CHROOT",
            "CAP_SYS_PTRACE"
        ],
        "ExecIDs": [
            "d8cf4c16f4ab9975e2d5c8666495738e44f0f68cf946396cf557649393f17992"
        ],
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/66f0d91d5544bf986ea3b2718597182ad6457fa1b47e67fdadac4ddd486346b3/diff:/var/lib/containers/storage/overlay/02654ec2cc842826a590877e2aeba7fb9d3555ff9ef4f4ea9687558bab306d47/diff:/var/lib/containers/storage/overlay/31e2ac8166e2deffa988fdec09da96d0d030448bc5a2385ee28ee551c7a4e1ec/diff:/var/lib/containers/storage/overlay/ebcd6ea000e6206dd669a6b5b983186ef9b7b5f8d9eb137443c10e1b201a7e95/diff:/var/lib/containers/storage/overlay/e542f72641415cf30c1af869cb49ce16a89e46db071bead7812ad15fbc3cb068/diff",
                "MergedDir": "/var/lib/containers/storage/overlay/f58bf45ec5ab9bd7c75b9a71a36a0e1560d63cdf263b93f51fa5cd93c249f3dc/merged",
                "UpperDir": "/var/lib/containers/storage/overlay/f58bf45ec5ab9bd7c75b9a71a36a0e1560d63cdf263b93f51fa5cd93c249f3dc/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/f58bf45ec5ab9bd7c75b9a71a36a0e1560d63cdf263b93f51fa5cd93c249f3dc/work"
            }
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "vscode-workspace-myproject",
                "Source": "/var/lib/containers/storage/volumes/vscode-workspace-myproject/_data",
                "Destination": "/workspace",
                "Driver": "local",
                "Mode": "",
                "Options": [
                    "nosuid",
                    "nodev",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "vscode",
                "Source": "/var/lib/containers/storage/volumes/vscode/_data",
                "Destination": "/vscode",
                "Driver": "local",
                "Mode": "",
                "Options": [
                    "nosuid",
                    "nodev",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Dependencies": [],
        "NetworkSettings": {
            "EndpointID": "",
            "Gateway": "10.88.0.1",
            "IPAddress": "10.88.0.34",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "96:73:da:99:24:e3",
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/run/netns/cni-d8ee9155-80ea-92eb-50e4-54fc1f589fb5",
            "Networks": {
                "podman": {
                    "EndpointID": "",
                    "Gateway": "10.88.0.1",
                    "IPAddress": "10.88.0.34",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "96:73:da:99:24:e3",
                    "NetworkID": "podman",
                    "DriverOpts": null,
                    "IPAMConfig": null,
                    "Links": null
                }
            }
        },
        "ExitCommand": [
            "/usr/bin/podman",
            "--root",
            "/var/lib/containers/storage",
            "--runroot",
            "/var/run/containers/storage",
            "--log-level",
            "info",
            "--cgroup-manager",
            "systemd",
            "--tmpdir",
            "/var/run/libpod",
            "--runtime",
            "crun",
            "--storage-driver",
            "overlay",
            "--storage-opt",
            "overlay.mountopt=nodev,metacopy=on",
            "--events-backend",
            "journald",
            "container",
            "cleanup",
            "bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90"
        ],
        "Namespace": "",
        "IsInfra": false,
        "Config": {
            "Hostname": "bd34f2e6887f",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "container=oci",
                "DISTTAG=f34container",
                "FGC=f34",
                "HOME=/root",
                "HOSTNAME=bd34f2e6887f"
            ],
            "Cmd": [
                "-c",
                "echo Container started ; trap \"exit 0\" 15; while sleep 1 \u0026 wait $!; do :; done"
            ],
            "Image": "localhost/vsc-devcontainer10-a57cf9f61fcb18f79530ad9aa379d3a9:latest",
            "Volumes": null,
            "WorkingDir": "/workspace",
            "Entrypoint": "/bin/sh",
            "OnBuild": null,
            "Labels": {
                "io.buildah.version": "1.19.4",
                "license": "MIT",
                "name": "fedora",
                "vendor": "Fedora Project",
                "version": "34",
                "vsch.local.folder": "/Users/christian/projects/devcontainer10",
                "vsch.quality": "stable",
                "vsch.remote.devPort": "0"
            },
            "Annotations": {
                "io.container.manager": "libpod",
                "io.kubernetes.cri-o.Created": "2021-03-19T14:58:40.933062845+01:00",
                "io.kubernetes.cri-o.TTY": "false",
                "io.podman.annotations.autoremove": "FALSE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.privileged": "FALSE",
                "io.podman.annotations.publish-all": "FALSE",
                "io.podman.annotations.seccomp": "unconfined",
                "org.opencontainers.image.stopSignal": "15"
            },
            "StopSignal": 15,
            "CreateCommand": [
                "/usr/local/Cellar/podman/3.0.1/bin/podman",
                "--log-level=error",
                "run",
                "--sig-proxy=false",
                "-a",
                "STDOUT",
                "-a",
                "STDERR",
                "--mount",
                "source=vscode-workspace-myproject,target=/workspace,type=volume",
                "--mount",
                "type=volume,src=vscode,dst=/vscode",
                "-l",
                "vsch.local.folder=/Users/christian/projects/devcontainer10",
                "-l",
                "vsch.quality=stable",
                "-l",
                "vsch.remote.devPort=0",
                "--cap-add=SYS_PTRACE",
                "--security-opt",
                "seccomp=unconfined",
                "--entrypoint",
                "/bin/sh",
                "vsc-devcontainer10-a57cf9f61fcb18f79530ad9aa379d3a9",
                "-c",
                "echo Container started ; trap \"exit 0\" 15; while sleep 1 \u0026 wait $!; do :; done"
            ],
            "Umask": "0022"
        },
        "HostConfig": {
            "Binds": [
                "vscode-workspace-myproject:/workspace:rw,rprivate,nosuid,nodev,rbind",
                "vscode:/vscode:rw,rprivate,nosuid,nodev,rbind"
            ],
            "CgroupManager": "systemd",
            "CgroupMode": "private",
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "k8s-file",
                "Config": null,
                "Path": "/var/lib/containers/storage/overlay-containers/bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90/userdata/ctr.log",
                "Tag": "",
                "Size": "0B"
            },
            "NetworkMode": "bridge",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [
                "CAP_SYS_PTRACE"
            ],
            "CapDrop": [
                "CAP_NET_RAW",
                "CAP_AUDIT_WRITE",
                "CAP_MKNOD"
            ],
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": [],
            "IpcMode": "private",
            "Cgroup": "",
            "Cgroups": "default",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "private",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "seccomp=unconfined"
            ],
            "Tmpfs": {},
            "UTSMode": "private",
            "UsernsMode": "",
            "ShmSize": 65536000,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": 0,
            "OomKillDisable": false,
            "PidsLimit": 2048,
            "Ulimits": [
                {
                    "Name": "RLIMIT_NOFILE",
                    "Soft": 1048576,
                    "Hard": 1048576
                },
                {
                    "Name": "RLIMIT_NPROC",
                    "Soft": 4194304,
                    "Hard": 4194304
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "CgroupConf": null
        }
    }
]

Any more ideas?

some more details:

interestingly running the server command manually doesn't lead to an error:

podman exec -ti bd34f2e6887f  /root/.vscode-server/bin/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8/server.sh --force-disable-user-env --port 0 --extensions-download-dir /root/.vscode-server/extensionsCache --start-server

*
* Visual Studio Code Server
*
* Reminder: You may only use this software with Visual Studio family products,
* as described in the license https://aka.ms/vscode-remote/license
*

Extension host agent listening on 41889

[14:14:06] Extension host agent started.

from the filesystem:

$ podman exec -ti bd34f2e6887f tree -a /tmp/
/tmp/
|-- vscode-remote-containers-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.js
`-- vscode-remote-containers-server-de3d75a6600fae3a7cd35bdfb8960be64ffe6559.js

$ podman exec -ti bd34f2e6887f tree -a /root
/root
|-- .bash_history
|-- .bash_logout
|-- .bash_profile
|-- .bashrc
|-- .cshrc
|-- .gnupg
|   |-- pubring.kbx
|   `-- trustdb.gpg
|-- .ssh
|   `-- known_hosts
|-- .tcshrc
|-- .viminfo
|-- .vscode-server
|   |-- bin
|   |   `-- 2b9aebd5354a3629c3aba0a5f5df49f43d6689f8 -> /vscode/vscode-server/bin/x64/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8
|   |-- data
|   |   `-- Machine
|   |       |-- .installExtensionsMarker
|   |       |-- .writeMachineSettingsMarker
|   |       `-- settings.json
|   `-- extensionsCache
|       |-- aws-scripting-guy.cform-0.0.24 -> /vscode/vscode-server/extensionsCache/aws-scripting-guy.cform-0.0.24
|       |-- coenraads.bracket-pair-colorizer-1.0.61 -> /vscode/vscode-server/extensionsCache/coenraads.bracket-pair-colorizer-1.0.61
|       |-- dbaeumer.vscode-eslint-2.1.14 -> /vscode/vscode-server/extensionsCache/dbaeumer.vscode-eslint-2.1.14
|       |-- dbaeumer.vscode-eslint-2.1.16 -> /vscode/vscode-server/extensionsCache/dbaeumer.vscode-eslint-2.1.16
|       |-- dbaeumer.vscode-eslint-2.1.17 -> /vscode/vscode-server/extensionsCache/dbaeumer.vscode-eslint-2.1.17
|       |-- ecmel.vscode-html-css-1.10.2 -> /vscode/vscode-server/extensionsCache/ecmel.vscode-html-css-1.10.2
|       |-- esbenp.prettier-vscode-5.9.2 -> /vscode/vscode-server/extensionsCache/esbenp.prettier-vscode-5.9.2
|       |-- golang.go-0.22.1 -> /vscode/vscode-server/extensionsCache/golang.go-0.22.1
|       |-- golang.go-0.23.0 -> /vscode/vscode-server/extensionsCache/golang.go-0.23.0
|       |-- golang.go-0.23.2 -> /vscode/vscode-server/extensionsCache/golang.go-0.23.2
|       |-- mrmlnc.vscode-duplicate-1.2.1 -> /vscode/vscode-server/extensionsCache/mrmlnc.vscode-duplicate-1.2.1
|       |-- ms-azuretools.vscode-docker-1.10.0 -> /vscode/vscode-server/extensionsCache/ms-azuretools.vscode-docker-1.10.0
|       |-- ms-python.python-2021.2.582707922 -> /vscode/vscode-server/extensionsCache/ms-python.python-2021.2.582707922
|       |-- ms-python.python-2021.2.636928669 -> /vscode/vscode-server/extensionsCache/ms-python.python-2021.2.636928669
|       |-- ms-toolsai.jupyter-2021.2.603412351 -> /vscode/vscode-server/extensionsCache/ms-toolsai.jupyter-2021.2.603412351
|       |-- ms-toolsai.jupyter-2021.3.619093157 -> /vscode/vscode-server/extensionsCache/ms-toolsai.jupyter-2021.3.619093157
|       |-- ms-vscode.cpptools-1.2.2 -> /vscode/vscode-server/extensionsCache/ms-vscode.cpptools-1.2.2
|       |-- oderwat.indent-rainbow-7.5.0 -> /vscode/vscode-server/extensionsCache/oderwat.indent-rainbow-7.5.0
|       |-- redhat.vscode-yaml-0.15.0 -> /vscode/vscode-server/extensionsCache/redhat.vscode-yaml-0.15.0
|       |-- redhat.vscode-yaml-0.16.0 -> /vscode/vscode-server/extensionsCache/redhat.vscode-yaml-0.16.0
|       `-- streetsidesoftware.code-spell-checker-1.10.2 -> /vscode/vscode-server/extensionsCache/streetsidesoftware.code-spell-checker-1.10.2
|-- anaconda-ks.cfg
|-- anaconda-post-nochroot.log
|-- anaconda-post.log
`-- original-ks.cfg

8 directories, 38 files

and when I try to connect to the created container I get:

[16 ms] Remote-Containers 0.163.2 in VS Code 1.54.3 (2b9aebd5354a3629c3aba0a5f5df49f43d6689f8).
[213 ms] Start: Resolving Remote
[218 ms] Setting up container: affectionate_ptolemy
[218 ms] Host: ssh://f1.tailscale.tk
[235 ms] Start: Run: podman inspect --type container affectionate_ptolemy
[790 ms] Start: Check Docker is running
[790 ms] Start: Run: podman version --format {{.Server.APIVersion}}
[1362 ms] Server API version: 3.0.0
[1363 ms] Start: Inspecting container
[1363 ms] Start: Run: podman inspect --type container bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90
[1929 ms] Start: Run: podman exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=aa8bf2f4-69eb-45bb-b8ee-34d4d4c6c9921616163232667 bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90 /bin/sh
[1931 ms] Start: Run in container: uname -m
[2699 ms] x86_64
[2700 ms] 
[2700 ms] Start: Run in container: (cat /etc/os-release || cat /usr/lib/os-release) 2>/dev/null
[2727 ms] NAME=Fedora
VERSION="34 (Container Image Prerelease)"
ID=fedora
VERSION_ID=34
VERSION_CODENAME=""
PLATFORM_ID="platform:f34"
PRETTY_NAME="Fedora 34 (Container Image Prerelease)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:34"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/34/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=34
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=34
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container
[2727 ms] 
[2728 ms] Start: Run in container: cat /etc/passwd
[2761 ms] Start: Run in container: test -d /root/.vscode-server
[2787 ms] 
[2787 ms] 
[2788 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.writeMachineSettingsMarker' ; } 2> /dev/null
[2820 ms] 
[2820 ms] 
[2820 ms] Exit code 1
[2821 ms] Start: Run in container: cat /root/.vscode-server/data/Machine/settings.json
[2848 ms] {
        "terminal.integrated.shell.linux": "/bin/bash",
        "go.useGoProxyToCheckForToolUpdates": false,
        "go.useLanguageServer": true,
        "go.gopath": "/root/go",
        "go.goroot": "/usr/lib/golang",
        "go.toolsGopath": "/root/go/bin"
}
[2849 ms] 
[2851 ms] Start: Run in container: test -d /root/.vscode-server/bin/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8
[2874 ms] 
[2875 ms] 
[2875 ms] Start: Run in container: test -d /vscode/vscode-server/bin/x64/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8
[2901 ms] 
[2901 ms] 
[2901 ms] Start: Run in container: touch '/vscode/vscode-server/bin/x64/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8'
[2901 ms] Start: Launching Remote-Containers helper.
[2902 ms] Start: Run: gpgconf --list-dir agent-extra-socket
[2932 ms] 
[2933 ms] 
[2933 ms] Start: Run in container: gpgconf --list-dir agent-socket
[2974 ms] /root/.gnupg/S.gpg-agent
[2975 ms] 
[2975 ms] Start: Run in container: gpgconf --list-dir homedir
[3012 ms] /root/.gnupg
[3012 ms] 
[3012 ms] Start: Run in container: ls '/root/.gnupg/private-keys-v1.d' 2>/dev/null
[3042 ms] 
[3042 ms] 
[3042 ms] Exit code 2
[3042 ms] Start: Run in container: mkdir -p -m 700 '/root/.gnupg'
[3069 ms] 
[3069 ms] 
[3069 ms] Start: Run in container: # Test for /root/.ssh/known_hosts and ssh
[3070 ms] Start: Run: gpgconf --list-dir homedir
[3095 ms] /root/.ssh/known_hosts exists
[3095 ms] 
[3095 ms] Exit code 1
[3095 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-03b80f45aac307c7bb31a21132a24d8762762ee3.js' >/tmp/vscode-remote-containers-03b80f45aac307c7bb31a21132a24d8762762ee3.js
[3147 ms] 
[3147 ms] 
[3147 ms] Start: Run in container: gpgconf --list-dir homedir
[3174 ms] /root/.gnupg
[3174 ms] 
[3174 ms] Start: Run in container: command -v git >/dev/null 2>&1 && git config --system credential.helper '!f() { /root/.vscode-server/bin/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8/node /tmp/vscode-remote-containers-03b80f45aac307c7bb31a21132a24d8762762ee3.js $*; }; f' || true
[3205 ms] 
[3205 ms] 
[3205 ms] Start: Run in container: cat <<'EOF-/tmp/vscode-remote-containers-server-03b80f45aac307c7bb31a21132a24d8762762ee3.js' >/tmp/vscode-remote-containers-server-03b80f45aac307c7bb31a21132a24d8762762ee3.js
[3205 ms] Start: Preparing Extensions
[3392 ms] 
[3392 ms] 
[3392 ms] Start: Run in container: # Test for /root/.gnupg/pubring.kbx and gpg
[3393 ms] Start: Run: podman exec -i -u root -e REMOTE_CONTAINERS_SOCKETS=["/tmp/vscode-ssh-auth-03b80f45aac307c7bb31a21132a24d8762762ee3.sock","/root/.gnupg/S.gpg-agent"] -e REMOTE_CONTAINERS_IPC=/tmp/vscode-remote-containers-ipc-03b80f45aac307c7bb31a21132a24d8762762ee3.sock bd34f2e6887f1deb6162c268837d59b405f782b93c81e32d7dad00ea85f8bc90 /root/.vscode-server/bin/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8/node /tmp/vscode-remote-containers-server-03b80f45aac307c7bb31a21132a24d8762762ee3.js
[3418 ms] /root/.gnupg/pubring.kbx exists
[3418 ms] 
[3418 ms] Exit code 1
[3418 ms] Start: Run in container: set -o noclobber ; mkdir -p '/root/.vscode-server/data/Machine' && { > '/root/.vscode-server/data/Machine/.installExtensionsMarker' ; } 2> /dev/null
[3451 ms] 
[3451 ms] 
[3451 ms] Exit code 1
[3451 ms] Start: Run in container: # Test for /root/.gnupg/trustdb.gpg and gpg
[3485 ms] /root/.gnupg/trustdb.gpg exists
[3485 ms] 
[3485 ms] Exit code 1
[3485 ms] Start: Run in container: test -d /root/.vscode-server/extensionsCache && ls /root/.vscode-server/extensionsCache || true
[3486 ms] Start: Run: gpg-connect-agent updatestartuptty /bye
[3520 ms] aws-scripting-guy.cform-0.0.24
coenraads.bracket-pair-colorizer-1.0.61
dbaeumer.vscode-eslint-2.1.19
ecmel.vscode-html-css-1.10.2
esbenp.prettier-vscode-6.3.1
golang.go-0.23.2
mrmlnc.vscode-duplicate-1.2.1
ms-azuretools.vscode-docker-1.11.0
ms-python.python-2021.3.658691958
ms-toolsai.jupyter-2021.3.619093157
ms-vscode.cpptools-1.2.2
oderwat.indent-rainbow-7.5.0
redhat.vscode-yaml-0.16.0
streetsidesoftware.code-spell-checker-1.10.2
[3520 ms] 
[3520 ms] Start: Run in container: test -d /vscode/vscode-server/extensionsCache && ls /vscode/vscode-server/extensionsCache || true
[3554 ms] aws-scripting-guy.cform-0.0.24
coenraads.bracket-pair-colorizer-1.0.61
dbaeumer.vscode-eslint-2.1.14
dbaeumer.vscode-eslint-2.1.16
dbaeumer.vscode-eslint-2.1.17
dbaeumer.vscode-eslint-2.1.19
ecmel.vscode-html-css-1.10.2
esbenp.prettier-vscode-5.9.2
esbenp.prettier-vscode-6.3.1
golang.go-0.22.1
golang.go-0.23.0
golang.go-0.23.2
mrmlnc.vscode-duplicate-1.2.1
ms-azuretools.vscode-docker-1.10.0
ms-azuretools.vscode-docker-1.11.0
ms-python.python-2021.2.582707922
ms-python.python-2021.2.636928669
ms-python.python-2021.3.658691958
ms-toolsai.jupyter-2021.2.603412351
ms-toolsai.jupyter-2021.3.619093157
ms-vscode.cpptools-1.2.2
oderwat.indent-rainbow-7.5.0
redhat.vscode-yaml-0.15.0
redhat.vscode-yaml-0.16.0
streetsidesoftware.code-spell-checker-1.10.2
[3554 ms] 
[3555 ms] Start: Run in container: for pid in `cd /proc && ls -d [0-9]*`; do { echo $pid ; readlink /proc/$pid/cwd ; readlink /proc/$pid/ns/mnt ; cat /proc/$pid/stat | tr "
[3689 ms] Terminating old extension host agent.
[3689 ms] Start: Run in container: kill -9 -1227
[3721 ms] 
[3721 ms] 
[3722 ms] Start: Starting VS Code Server
[3722 ms] Start: Run in container: /root/.vscode-server/bin/2b9aebd5354a3629c3aba0a5f5df49f43d6689f8/server.sh --force-disable-user-env --port 0 --extensions-download-dir /root/.vscode-server/extensionsCache --start-server
[4271 ms] Remote-Containers server: undefined:1
[/tmp/vscode-ssh-auth-03b80f45aac307c7bb31a21132a24d8762762ee3.sock,/root/.gnupg/S.gpg-agent]
 ^

SyntaxError: Unexpected token / in JSON at position 1
    at JSON.parse (<anonymous>)
    at Object.<anonymous> (/tmp/vscode-remote-containers-server-03b80f45aac307c7bb31a21132a24d8762762ee3.js:6:6688)
    at Module._compile (internal/modules/cjs/loader.js:1137:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)
    at Module.load (internal/modules/cjs/loader.js:985:32)
    at Function.Module._load (internal/modules/cjs/loader.js:878:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47
[4283 ms] Error: stream ended with:0 but wanted:9
        at u (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:7:57762)
        at /Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:7:57943
        at s (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:12:5707)
        at Socket.<anonymous> (/Users/christian/.vscode/extensions/ms-vscode-remote.remote-containers-0.163.2/dist/extension/extension.js:12:5877)
        at Socket.emit (events.js:327:22)
        at endReadableNT (_stream_readable.js:1220:12)
        at processTicksAndRejections (internal/process/task_queues.js:84:21)
[4284 ms] Remote-Containers server terminated (code: 1, signal: null).

I think the problem is that $@ in ssh -T f1.tailscale.tk -o LogLevel=QUIET "podman $@" does not preserve the parameter list, but passes "podman $@" as a single parameter to ssh and the shell on the remote machine then can't split that into the original parameters because the parameters contain whitespace. Our code relies on the parameters being preserved.

Bit of an aside, but would it be possible to have some documentation on this, ergo, using a combination of a remote host and attaching to a container, and especially for podman in lieu of docker?

This is all great stuff and a bit black magic in a way- there's a lot under the hood and, not having used VS Code in a few years, the way it just creates and sorts out a development container with all of .net installed is certainly impressive. I liked it as an IDE before, but a few years ago I was finding some language plugins were a bit immature, and stuff like this is enough to tempt me back for the future. If I can just open an IDE locally and debug an application and dependencies for a language I don't know on a remote server running inside a container as if it were just on my machine, that's seriously impressive stuff, so good work - just unfortunate I still can't quite work out how to do it after ~5 hours.

I think there are a lot of variables and like many (and it would seem from the comments, like you too, Christof), I'm new to the podman world and am hitting a tonne of errors at different levels trying to get this to work. Many articles online mention a totally different way of doing things - e.g. loopback of the Docker socket or creating a context, that don't apply for podman, and I believe if I read correctly Code now uses the CLI so these aren't even needed.

If someone could consider documenting points like the below, then I'd be really grateful, as it seems other folks will be arriving at this situation increasingly over time as Podman/OCI take hold, and it looks like almost everything is there under the hood but as there are so many ways to go and as the technologies are both evolving quickly it's hard to know what I need to fix to have this working:

What is the simplest/best order of steps needed to debug an application running inside a container on a remote host? Do we open a window in SSH to the server and then attach to/create a container if the CLI is being used? Do we need to have podman-remote connected successfully and then go via calling the local binaries? Does the docker.host variable need to be set to point locally or remotely? Does the container need to be created already to appear? Or will Code create this and start remotely?

Thanks a lot :)

We have our documentation at https://code.visualstudio.com/docs/remote/containers. This does not include any documentation specific to Podman, but focuses on Docker.

So, I need to connect using the remote-containers features first and foremost, and forget using the remote-SSH beforehand?

I've been following this guide this afternoon: Developing inside a container on a remote Docker host but I'm still hitting several problems:

Have podman on both machines, but do not set up podman-remote
Ensure my key for this server is added to my ssh-agent source

Change settings.json to include the following:

"remote.containers.dockerPath": "podman",
"docker.host":"ssh://user@host" // This is greyed-out and apparently not recognised, but is having a difference somewhere.

Create a new scratch folder each time with an empty file to ensure that there are no cache problems (source I) - then choose a basic alpine image for the container for speed.
Build process returns an error ENONET - "is docker installed?"
- Create a symlink on the remote server to podman as /usr/local/bin/docker
Close and re-open Code window
Error - permission denied whilst copying
- edit devcontainer.json to comment out remoteUser: "vscode". Being root inside a rootless podman container means we have the host-side ID of the host user, and so is probably the correct way to handle in many (but not all) containers
Close and open again
The container is running successfully and I can open a terminal. The title bar displays [Dev Container: alpine @ ssh://user@host]. However the container is started on my local machine. I don't see any evidence on the server that images have been pulled etc., so it looks like the whole process has taken place on my localhost.

I can see near the top of the log, some commands are interacting with the local podman - this may of course be by design, but I know I'm using a newer version locally as my server is still running bionic.:

[92 ms] Remote-Containers 0.183.0 in VS Code 1.57.1 (507ce72a4466fbb27b715c3722558bb15afa9f48).
[91 ms] Start: Resolving Remote
[103 ms] Setting up container for folder or workspace: /mnt/ramdisk/containers/beta
[104 ms] Host: ssh://user@host
[106 ms] Start: Check Docker is running
[106 ms] Start: Run: podman version --format {{.Server.APIVersion}}
[417 ms] Server API version: 3.2.0-rc3

It seems like there are multiple places where this is doing things a bit differently than it's meant to be - unless I'm being a bit thick and am missing something. Am more than happy to contribute documentation (a skeleton at the least) if and when I get there, but it seems I'm still missing something to get this working on podman. Also, whilst a great reference, the article we both linked is quite long, and I think as a common use-case, a briefer tutorial for the basics of creating/attaching to a remote container would be a help to get people up and running; they could then be referred to the existing one for troubleshooting.

Thanks

The "docker.host" setting is passed to the command using the DOCKER_HOST environment variable. That might not be supported by Podman.

If Podman-Remote is a Docker-compatible CLI connecting to a remote server, that should work. I imagine you would first setup Podman-Remote to connect to the server you want and then let Remote-Containers use that. Let me know if that's how it works.

if Podman-Remote is a Docker-compatible CLI connecting to a remote server, that should work.

podman supports remote connections over ssh, but there's a bug that lets hang exec -i commands over ssh. (You can also use an original docker cli tool against the Podman api/socket, but it's affected by the same bug).

I tried to work around it with a wrapper (see above) and got a bit further than what I posted here, but ran into more issues. (Unfortunately it's been a while and I lost my notes about it. When I find some time I'll try again and update this issue).

That's okay, if I'm doing everything as I should be then I'll take a look into it when I can. Didn't realise the host was just simple environment variables, so can try going forward again with the wrapper. For the time being I've installed docker on a test machine but would be good to have working as I'm deploying with podman eventually so it's lighter. Would be good to have it documented when appears to be working, but in the meantime am happy to share notes etc. I ran into the exec issue with the pipes too. I also had issues connecting different versions of podman, I think it's changed somewhat between 3.0<-->3.1<-->3.2, so some of the issues may have ironed themselves out - but I couldn't verify easily as they weren't talking to each other. Podman-in-Podman didn't quite work for testing purposes, but if I set up a dedicated VM so we have the same version on both sides then I may be able to try with the above (not read and digested it all yet) and see if it's any further forward.

I think the problem is that $@ in ssh -T f1.tailscale.tk -o LogLevel=QUIET "podman $@" does not preserve the parameter list, but passes "podman $@" as a single parameter to ssh and the shell on the remote machine then can't split that into the original parameters because the parameters contain whitespace. Our code relies on the parameters being preserved.

I use fish these days, so my Bash may be a bit rusty, and SSH adds some complication, but I believe the correct thing would be:

ssh -T host -ologlevel=QUIET podman "$@" because ssh -T host -ologlevel=QUIET "podman $@" includes the command in the quotes so it is not split - podman is going to be run together with the arguments and usch a file does not exist, and ssh -T host -ologlevel=QUIET podman $* does not preserve whitespace in the first place so the argument split is split over the spaces.

Am just about to try some of this out and see.

So, the below is actually getting quite far for me as a wrapper (Focal and podman 3.2.2 on both sides):

podman-remote --connection "$DOCKER_HOST" "$@"

The exec | md5sum loop was stable even if slow at times over SSH, although am using a VM so may be part of it. It would appear to hang but eventually continued, then was fine. And it importantly brought back the same hash.

In order for the above to work, I simply set the docker.host in the settings to match the connection name for podman-remote:

profile=profileName
connection=ssh://user@host:/run/user/1000/podman/podman.sock
key=~/.ssh/myKey

podman-remote system connection add $profile --identity $key $connection

It would be more sophisticated to parse the SSH url and match against the connections, but given it's a bit of a settings override this was fine and much easier.

As my podman is running as rootless, I comment out the username in devcontainer.json so that it runs as root, which translates myself on the remote machine. Otherwise I think it fails to elevate inside the container at the point it creates/root/.vscode-server

Now the issue is that it is cycling round and round trying to open some ports. I imagine because podman's pods work differently regarding network space, and/or I know that ports can't be opened on a pod once created, the pod must be recreated, and IIRC there may be workarounds for this in Docker:

[35416 ms] Port forwarding connection from 33276 > 37489 > 37489 in the container.
[35417 ms] Start: Run in container: /root/.vscode-server/bin/2d23c42a936db1c7b3b06f918cde29561cc47cd6/node -e 
[36149 ms] [16:25:00] [::ffff:127.0.0.1][6861316d][ManagementConnection] The client has reconnected.
[36149 ms] Port forwarding 33270 > 37489 > 37489 stderr: Remote close
[36192 ms] Port forwarding 33270 > 37489 > 37489 terminated by extension with code null and signal SIGKILL.
[39731 ms] Port forwarding 33276 > 37489 > 37489: Local close
[39739 ms] Start: Run: /mnt/ramdisk/wrapper.sh inspect --type container 2dfeb99f303360d714d36ca27f6e39d216777d6cf7635f43d507336b583e2d4b
[40362 ms] Port forwarding connection from 33282 > 37489 > 37489 in the container.
[40362 ms] Start: Run in container: /root/.vscode-server/bin/2d23c42a936db1c7b3b06f918cde29561cc47cd6/node -e 
[41074 ms] [16:25:05] [::ffff:127.0.0.1][6861316d][ManagementConnection] The client has reconnected.
[41078 ms] Port forwarding 33276 > 37489 > 37489 stderr: Remote close
[41120 ms] Port forwarding 33276 > 37489 > 37489 terminated by extension with code null and signal SIGKILL.
[44731 ms] Port forwarding 33282 > 37489 > 37489: Local close
[44773 ms] Start: Run: /mnt/ramdisk/wrapper.sh inspect --type container 2dfeb99f303360d714d36ca27f6e39d216777d6cf7635f43d507336b583e2d4b

So I don't know what to do about that right now but that would probably be the immediate next step. VS Code can connect to a running container to start its server and then attach as a debugger, so am I right in thinking that above where it's starting the node process it's expecting some ability to find an unused port and expose it to the host dynamically?

We are using docker exec to access the container and for forwarding the port inside the container. That should all be possible dynamically. Not sure why the connection is closed, but it seems to be closed on the container's side first (Remote close) after being successful to some degree (The client has reconnected.).

How is it that the port is exposed using exec? Or is it using some sort of trick like can be achieved with an SSH loopback? I thought generally once a container is up, new ports can't be exposed.

12 Jul 2021 14:59:30 Christof Marti @.***>:

We are using docker exec to access the container and for forwarding the port inside the container. That should all be possible dynamically. Not sure why the connection is closed, but it seems to be closed on the container's side first (Remote close) after being successful to some degree (The client has reconnected.).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub[https://github.com/microsoft/vscode-remote-release/issues/4643#issuecomment-878303002], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ABGF2AJ23VMVJZ2DO5UYPWLTXLYMFANCNFSM4Y6MF42A]. [data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEgAAABICAYAAABV7bNHAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAArSURBVHic7cEBDQAAAMKg909tDjegAAAAAAAAAAAAAAAAAAAAAAAAAAA+DFFIAAEctgHwAAAAAElFTkSuQmCC###24x24:true###][Tracking image][https://github.com/notifications/beacon/ABGF2APY6FPNEO7UUBSMUXDTXLYMFA5CNFSM4Y6MF42KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGRM5OGQ.gif]

The Remote-Containers extension listens on a local port and when VS Code connects, it uses docker exec to connect to the port in the container. (VS Code itself does not have any container-related code, only the extension has.)

Right okay, so the container is the client and the local machine is the server in effect. In that's case I'll have to have a more detailed look.

12 Jul 2021 17:08:15 Christof Marti @.***>:

The Remote-Containers extension listens on a local port and when VS Code connects, it uses docker exec to connect to the port in the container. (VS Code itself does not have any container-related code, only the extension has.)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub[https://github.com/microsoft/vscode-remote-release/issues/4643#issuecomment-878405556], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ABGF2AMIYYBWOD7GP3WJW33TXMHO7ANCNFSM4Y6MF42A]. [data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEgAAABICAYAAABV7bNHAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAArSURBVHic7cEBDQAAAMKg909tDjegAAAAAAAAAAAAAAAAAAAAAAAAAAA+DFFIAAEctgHwAAAAAElFTkSuQmCC###24x24:true###][Tracking image][https://github.com/notifications/beacon/ABGF2AOBRUUOJQOSB7DPBVDTXMHO7A5CNFSM4Y6MF42KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOGRNWPNA.gif]

Just to clarify: The VS Code UI on your local machine connects to a port on localhost that then uses docker exec to connect to the container and in the container connects to a port of the VS Code Server running inside the container. docker exec's standard in and out are used to transfer the connection data.

I'll give it a go when I have a spare minute.

Probably fixed in this PR (https://github.com/containers/podman/pull/11503). I was able to attach to a DevContainer with Remote Containers from macOS!

VS Code

VS Code: Version: v1.60.0
Remote - Containers: v0.194.0

Server side:

$ systemctl --user enable --now podman.socket
$ podman version
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.16.6
Git Commit:   63f6656f8fa79d7f6e01379d7ba0aa4ab3c03b37
Built:        Fri Sep 10 08:50:22 2021
OS/Arch:      linux/amd64

macOS side:

$ podman system connection add fedora-test --identity ~/.ssh/id_ecdsa ssh://user@192.168.1.32:22
$ podman system connection default fedora-test
$ podman version
Client:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.17
Built:        Tue Aug 31 04:15:26 2021
OS/Arch:      darwin/amd64

Server:
Version:      4.0.0-dev
API Version:  4.0.0-dev
Go Version:   go1.16.6
Git Commit:   63f6656f8fa79d7f6e01379d7ba0aa4ab3c03b37
Built:        Fri Sep 10 08:50:22 2021
OS/Arch:      linux/amd64
$ podman run --rm -d --name ubi ubi8 sleep inf

Thanks for the update! Closing as fixed upstream.

Just an FYI, I saw this in the logs for a local container: WARN[0001] SHELL is not supported for OCI image format, [/bin/sh -c] will be ignored. Must usedockerformat Sounded familiar to this issue - will update to podman 4 when I can, but may be a while.

When I can update, I'll keep an eye out for any similar messages and update if there's anything outstanding or warnings that ought to be taken into account for podman/OCI support going forward.

microsoft / vscode-remote-release

remote containers - podman over ssh not working #4643