search

gitpod-io / gitpod

The developer platform for on-demand cloud development environments to create software faster and more securely.
https://www.gitpod.io
GNU Affero General Public License v3.0
12.91k stars 1.24k forks source link

[Self-Hosted] Custom docker image build failure : 'failed to mount /tmp/containerd-mount<XXXXXXXX>: operation not permitted' #9188

Closed ccll closed 2 years ago

ccll commented 2 years ago

Bug description

My .gitpod.Dockerfile builds successfully with running docker build ... directly in workspace container, but failed to build after commited to git and started a new workspace based on that new commit.

Related logs:

...
.gitpod.Dockerfile:3
--------------------
   1 |     FROM gitpod/workspace-python:latest
   2 |     
   3 | >>> RUN pyenv install 3.10.2
   4 |     RUN pyenv global 3.10.2
   5 |     
--------------------
error: failed to solve: failed to compute cache key: failed to mount /tmp/containerd-mount2739100871: operation not permitted
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","command":"build","error":"exit status 1","level":"error","message":"build failed","serviceContext":{"service":"bob","version":""},"severity":"ERROR","time":"2022-04-08T01:05:21Z"}
exit
...

Steps to reproduce

My host environment:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

(I know that only Ubuntu is officially supported, but my host is a legacy one and are running many production services on it now, and since Gitpod can almost run without much problem on it, I think there might be a way to make Debian as a community supported distro for Gitpod.)

Installed kernel 5.10 from buster-backports.

$ uname -a
Linux homelab-1 5.10.0-0.bpo.12-amd64 #1 SMP Debian 5.10.103-1~bpo10+1 (2022-03-08) x86_64 GNU/Linux

Kubernetes is a single node k3s cluster.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4+k3s1", GitCommit:"43b1cb48200d8f6af85c16ed944d68fcc96b6506", GitTreeState:"clean", BuildDate:"2022-02-24T22:38:17Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4+k3s1", GitCommit:"43b1cb48200d8f6af85c16ed944d68fcc96b6506", GitTreeState:"clean", BuildDate:"2022-02-24T22:38:17Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
$ crictl info
{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "/var/lib/rancher/k3s/data/4012316506613ee8c3cffc1e5b5eca706270685d33585804b257e93ea98d1917/bin"
    ],
    "PluginConfDir": "/var/lib/rancher/k3s/agent/etc/cni/net.d",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "cni-loopback",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "loopback",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"type\":\"loopback\"}"
            }
          ],
          "Source": "{\n\"cniVersion\": \"0.3.1\",\n\"name\": \"cni-loopback\",\n\"plugins\": [{\n  \"type\": \"loopback\"\n}]\n}"
        },
        "IFName": "lo"
      },
      {
        "Config": {
          "Name": "cbr0",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "flannel",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"delegate\":{\"forceAddress\":true,\"hairpinMode\":true,\"isDefaultGateway\":true},\"type\":\"flannel\"}"
            },
            {
              "Network": {
                "type": "portmap",
                "capabilities": {
                  "portMappings": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"portMappings\":true},\"type\":\"portmap\"}"
            }
          ],
          "Source": "{\n  \"name\":\"cbr0\",\n  \"cniVersion\":\"0.3.1\",\n  \"plugins\":[\n    {\n      \"type\":\"flannel\",\n      \"delegate\":{\n        \"hairpinMode\":true,\n        \"forceAddress\":true,\n        \"isDefaultGateway\":true\n      }\n    },\n    {\n      \"type\":\"portmap\",\n      \"capabilities\":{\n        \"portMappings\":true\n      }\n    }\n  ]\n}\n"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "overlayfs",
      "defaultRuntimeName": "runc",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "runtimes": {
        "runc": {
          "runtimeType": "io.containerd.runc.v2",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "ContainerAnnotations": null,
          "runtimeRoot": "",
          "options": null,
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": ""
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": true,
      "discardUnpackedLayers": false
    },
    "cni": {
      "binDir": "/var/lib/rancher/k3s/data/4012316506613ee8c3cffc1e5b5eca706270685d33585804b257e93ea98d1917/bin",
      "confDir": "/var/lib/rancher/k3s/agent/etc/cni/net.d",
      "maxConfNum": 1,
      "confTemplate": ""
    },
    "registry": {
      "configPath": "",
      "mirrors": null,
      "configs": null,
      "auths": null,
      "headers": null
    },
    "imageDecryption": {
      "keyModel": "node"
    },
    "disableTCPService": true,
    "streamServerAddress": "127.0.0.1",
    "streamServerPort": "10010",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 1024,
    "sandboxImage": "rancher/mirrored-pause:3.6",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 16384,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": true,
    "disableHugetlbController": true,
    "ignoreImageDefinedVolumes": false,
    "netnsMountsUnderStateDir": false,
    "containerdRootDir": "/var/lib/rancher/k3s/agent/containerd",
    "containerdEndpoint": "/run/k3s/containerd/containerd.sock",
    "rootDir": "/var/lib/rancher/k3s/agent/containerd/io.containerd.grpc.v1.cri",
    "stateDir": "/run/k3s/containerd/io.containerd.grpc.v1.cri"
  },
  "golang": "go1.17.5",
  "lastCNILoadStatus": "OK"

Gitpod installer config file:

apiVersion: v1
authProviders: []
blockNewUsers:
  enabled: false
  passlist: []
certificate:
  kind: secret
  name: https-certificates
containerRegistry:
  inCluster: true
database:
  inCluster: true
disableDefinitelyGp: false
domain: "<my.domain.com>"
kind: Full
metadata:
  region: local
objectStorage:
  inCluster: true
observability:
  logLevel: info
openVSX:
  url: https://open-vsx.org
repository: eu.gcr.io/gitpod-core-dev/build
workspace:
  resources:
    requests:
      cpu: "1"
      memory: 2Gi
  runtime:
    containerdRuntimeDir: /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io
    containerdSocket: /run/k3s/containerd/containerd.sock
    fsShiftMethod: fuse

.gitpod.yml

image:
  file: .gitpod.Dockerfile

tasks:
  - init: echo hello

.gitpod.Dockerfile

FROM gitpod/workspace-python:latest

RUN pyenv install 3.10.2
RUN pyenv global 3.10.2

Workspace affected

No response

Expected behavior

The docker image should build successfully just like running docker build ... in workspace container.

Example repository

No response

Anything else?

No response

shaoye commented 2 years ago

same issue here, did you solve it?

ccll commented 2 years ago

@shaoye Nope, still waiting the canonical solution.

Right now I workaround this by manually docker build & docker push to a container registry and reference the pushed image there.

Seenox commented 2 years ago

Same issue here and the aforementioned workaround wouldn't work for our use cases. Could someone from GitPod Team help here, please?

shaoye commented 2 years ago

we switched to Ubuntu 2004 and it worked

mrzarquon commented 2 years ago

@ccli & @seenox what is your containerd version (ctr version)?

I'm tracking down info so we can get a resolution on this.

ccll commented 2 years ago

@mrzarquon Sorry for the late reply. Here is my ctr version

Client:
  Version:  1.4.4
  Revision: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
  Go version: go1.13.15

Server:
  Version:  1.4.4
  Revision: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
  UUID: 8c3f8d28-fa9a-429b-8a5c-ee3d7954b79c
utam0k commented 2 years ago

@ccll Hi, thanks for your information. Can you open this repository in your gitpod? https://github.com/utam0k/gitpod-playground/tree/eStargz https://${your_gitpod}/#https://github.com/utam0k/gitpod-playground/tree/eStargz

utam0k commented 2 years ago

This issue is brought into by eStargz. Fortunately, we just removed it. https://github.com/gitpod-io/gitpod/pull/9492

Seenox commented 2 years ago

@mrzarquon @utam0k sorry for the late reply (and bumping the issue). Here is my ctr version:

Client:
  Version:  1.5.4
  Revision: 69a4a55aa42353c71aae40e5bb27a412d8e3149b
  Go version: go1.13.15

Server:
  Version:  v1.4.6
  Revision: d71fcd7d8303cbf684402823e425e9dd2e99285d
  UUID: e092e812-9a86-44ad-b6b0-dbec0f674629

And I'm not able to open the example repository. Checked in v2022.4.2, installed using kots.

utam0k commented 2 years ago

@Seenox cc: @gitpod-io/engineering-self-hosted Thanks for your confirmation. It's turned out that the version of contained is probably irrelevant. The main branch has already included fixed PR. So, please wait for releasing the latest self-hosted version. If you will get the same error with the latest main branch, please feel free to reopen this issue.

ccll commented 2 years ago

Confirmed the issue was fixed in release-2022.05.0.5. :heart:

Seenox commented 2 years ago

Also confirming the issue has been fixed. Thanks!