[fusefs]: docker in workspace not working

mrsimonemms commented 2 years ago

Bug description

It seems that running docker run -it --rm <container> is returning the dreaded proc error again when deployed with the Installer. Thanks to @csweichel's suggestion, it seems to be to work with the fsShiftMethod - if you set to fuse (the default) then it fails. Setting to shiftfs makes it work

This is NOT an issue in the SaaS offering.

Steps to reproduce

Run docker run -it --rm node on a GKE or Azure cluster. Must have workspace.runtime.fsShiftMethod set to fuse - setting to shiftfs seems to work

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

ws-daemon logs (redacted)

Azure

{"args":["/app/nsinsider","move-mount","--target","/tmp/supervisor1525920107/workspace/.docker-root/overlay2/b32db367d8b8fb33ad8258da860fdaba48805dba17e874051a96d1619b2d1967/merged/proc","--pipe-fd","3"],"instanceId":"c55dbb52-b779-45a5-8a7d-4af9a43bc38f","level":"fatal","message":"no such file or directory","serviceContext":{"service":"nsinsider","version":""},"severity":"CRITICAL","time":"2021-12-01T12:16:39Z"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"cannot move mount:\n    github.com/gitpod-io/gitpod/ws-daemon/pkg/iws.moveMount\n        github.com/gitpod-io/gitpod/ws-daemon/pkg/iws/iws.go:601\n  - cannot run nsinsider:\n    github.com/gitpod-io/gitpod/ws-daemon/pkg/iws.nsinsider\n        github.com/gitpod-io/gitpod/ws-daemon/pkg/iws/iws.go:718\n  - exit status 1","instanceId":"c55dbb52-b779-45a5-8a7d-4af9a43bc38f","level":"error","message":"cannot mount proc","procPID":21447,"reqPID":3612,"serviceContext":{"service":"ws-daemon","version":"commit-1c0c677313692945976566d0d042e4f033889de2"},"severity":"ERROR","time":"2021-12-01T12:16:39Z","userId":"5d18ef9f-0816-4697-890d-d8107eb9efc0","workspaceId":"coffee-aardwolf-jz9bzds6"}

GCP

{"args":["/app/nsinsider","move-mount","--target","/tmp/supervisor3137808028/workspace/.docker-root/overlay2/d1d0c69f966609211dc2325e8a18c4ae9631d9121e4eff04083224e552173fcb/merged/proc","--pipe-fd","3"],"instanceId":"16774d36-b3c2-4565-8e2c-c71aef4d52e3","level":"fatal","message":"no such file or directory","serviceContext":{"service":"nsinsider","version":""},"severity":"CRITICAL","time":"2021-12-01T13:57:12Z"}
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"cannot move mount:\n    github.com/gitpod-io/gitpod/ws-daemon/pkg/iws.moveMount\n        github.com/gitpod-io/gitpod/ws-daemon/pkg/iws/iws.go:601\n  - cannot run nsinsider:\n    github.com/gitpod-io/gitpod/ws-daemon/pkg/iws.nsinsider\n        github.com/gitpod-io/gitpod/ws-daemon/pkg/iws/iws.go:718\n  - exit status 1","instanceId":"16774d36-b3c2-4565-8e2c-c71aef4d52e3","level":"error","message":"cannot mount proc","procPID":134499,"reqPID":2074,"serviceContext":{"service":"ws-daemon","version":"commit-1c0c677313692945976566d0d042e4f033889de2"},"severity":"ERROR","time":"2021-12-01T13:57:12Z","userId":"3f6e8165-7ffa-4928-b4cb-75be9cc98099","workspaceId":"salmon-fish-r7hguwy6"}

To my observation, these seem to be the same basic problem between Azure and GCP

csweichel commented 2 years ago

/schedule

roboquat commented 2 years ago

@csweichel: Issue scheduled in the workspace team (WIP: 0)

In response to [this](https://github.com/gitpod-io/gitpod/issues/6992#issuecomment-983597495): >/schedule Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

szab100 commented 2 years ago

@MrSimonEmms Thanks, verified to be working on gke-guide based deployments:

After adding yq e -i '.workspace.runtime.fsShiftMethod = "shiftfs"' "${CONFIG_FILE}" to setup.sh & reinstall:

gitpod /workspace/template-typescript-react $ docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:latest
Unable to find image 'mysql:latest' locally
latest: Pulling from library/mysql
a10c77af2613: Pull complete 
b76a7eb51ffd: Pull complete 
258223f927e4: Pull complete 
2d2c75386df9: Pull complete 
63e92e4046c9: Pull complete 
f5845c731544: Pull complete 
bd0401123a9b: Pull complete 
3ef07ec35f1a: Pull complete 
c93a31315089: Pull complete 
3349ed800d44: Pull complete 
6d01857ca4c1: Pull complete 
4cc13890eda8: Pull complete 
Digest: sha256:aeecae58035f3868bf4f00e5fc623630d8b438db9d05f4d8c6538deb14d4c31b
Status: Downloaded newer image for mysql:latest
b5349ce3f8b4e081bd728f76b171b9667e90ca1190c9863702b795eb2447cb74
gitpod /workspace/template-typescript-react $ docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS         PORTS                 NAMES
b5349ce3f8b4   mysql:latest   "docker-entrypoint.s…"   2 seconds ago   Up 2 seconds   3306/tcp, 33060/tcp   some-mysql

csweichel commented 2 years ago

utam0k commented 2 years ago

I made sure this issue related to the libseccomp issue. #7793 When I applied the libseccomp version upgrade, docker worked fine. It will not work with core-dev because the linux kernel version must not be above 5.6. So resolving issue #7793 will automatically fix this issue as well.

mrsimonemms commented 2 years ago

I've recently switched the default config method to shiftfs so closing - see 5f9f701835292bbe28e71c8eb70f8a09965cb5ab

gitpod-io / gitpod