search

GoogleContainerTools / kaniko

Build Container Images In Kubernetes
Apache License 2.0
14.76k stars 1.43k forks source link

GCR + GCB "runtime: out of memory" race condition #788

Open ptinsley opened 5 years ago

ptinsley commented 5 years ago

Actual behavior Adding build log in a comment, you get an out of memory error

Expected behavior Either consistently work or consistently fail (due to a bad build)

To Reproduce Steps to reproduce the behavior:

  1. Submit builds to GCB

builds fail randomly, hit the retry button on a build it will likely succeed the next time.

Additional Information

bring kaniko bits over from the kankio container

COPY --from=gcr.io/kaniko-project/executor /kaniko /

COPY cb/setup_env /tmp/cb/ COPY requirements.txt /tmp/

RUN /tmp/cb/setup_env && rm -rf /tmp/*

Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
 - Build Context
requirements.txt
```slackclient>=2.0.0
click
pyyaml
PyGithub
pre-commit
cryptography
ruamel.yaml
tenacity

setup_env

#!/usr/bin/env bash
# code: language=bash insertSpaces=true tabSize=2
set -ex

PYTHON_VERSION=3.7

script_path="$( cd "$(dirname "$0")" ; pwd -P )"
#DO NOT call core library, it will make an infinite loop (it calls this script if python isn't happy)

#if this was called, we probably need to install *something* go ahead and do an apt update
apt-get update -qq

#add some pre-reqs
apt-get install -f -y \
curl \
lsb-release \
apt-utils \
software-properties-common \
gnupg2

if [[ `which python${PYTHON_VERSION}` ]]; then
    echo "Detected python${PYTHON_VERSION} already installed"
else
    echo "No python${PYTHON_VERSION}, attempting to install (only setup to do this for apt based distros)"
    apt-get install -y software-properties-common curl
    add-apt-repository --yes ppa:deadsnakes/ppa

    apt-get update -qq
    apt-get install -y python$PYTHON_VERSION-dev python3-distutils

    curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
    python$PYTHON_VERSION get-pip.py && rm get-pip.py

    python$PYTHON_VERSION --version
fi

if [[ `which docker` ]]; then
    echo "Detected docker executable"
else
    echo "No docker executable installed, installing."
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
    add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
    apt-get update -qq
    apt-get install -y docker-ce-cli
    docker -v
fi

if [[ `which git` ]]; then
    echo "Detected git executable"
else
    echo "No git executable installed, installing."
    apt-get install -f -y git
    git --version
fi

if [[ `which berglas` ]]; then
    echo "Detected berglas executable"
else
    echo "No berglas executable installed, installing."
    apt-get install -f -y curl
    curl https://storage.googleapis.com/berglas/master/linux_amd64/berglas -o /bin/berglas
    chmod 755 /bin/berglas
fi

if [[ `which gcloud` ]]; then
    echo "Detected gcloud executable"
else
    echo "deb http://packages.cloud.google.com/apt cloud-sdk-$(lsb_release -c -s) main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
    curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
    apt-get update -qq
    apt-get install -f -y google-cloud-sdk kubectl
    gcloud components list
    gsutil -v
fi

if hash pip3 2> /dev/null; then
    echo "Detected pip3 already installed"
else
    echo "No pip3, attempting to install (only setup to do this for apt based distros)"
    apt-get install -f -y python3-pip
    pip3 --version
fi

echo "installing python modules"
ls -l
pip3 install -r $script_path/../requirements.txt
ptinsley commented 5 years ago 
BUILD
Starting Step #0 - "Build Base"
Step #0 - "Build Base": Pulling image: gcr.io/kaniko-project/executor
Step #0 - "Build Base": Using default tag: latest
Step #0 - "Build Base": latest: Pulling from kaniko-project/executor
Step #0 - "Build Base": Digest: sha256:265014d50d302ab65c2fd58f2cc8a2c577abb5acdb7cf422e88974a68ce403b2
Step #0 - "Build Base": Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
Step #0 - "Build Base": gcr.io/kaniko-project/executor:latest
Step #0 - "Build Base": INFO[0000] Resolved base name ubuntu:18.04 to ubuntu:18.04 
Step #0 - "Build Base": INFO[0000] Downloading base image gcr.io/kaniko-project/executor 
Step #0 - "Build Base": INFO[0000] Storing source image from stage gcr.io/kaniko-project/executor at path /kaniko/stages/gcr.io/kaniko-project/executor 
Step #0 - "Build Base": INFO[0003] Resolved base name ubuntu:18.04 to ubuntu:18.04 
Step #0 - "Build Base": INFO[0003] Downloading base image ubuntu:18.04          
Step #0 - "Build Base": INFO[0004] Error while retrieving image from cache: getting file info: stat /cache/sha256:1bbdea4846231d91cce6c7ff3907d26fca444fd6b7e3c282b90c7fe4251f9f86: no such file or directory 
Step #0 - "Build Base": INFO[0004] Downloading base image ubuntu:18.04          
Step #0 - "Build Base": INFO[0005] Built cross stage deps: map[]                
Step #0 - "Build Base": INFO[0005] Downloading base image ubuntu:18.04          
Step #0 - "Build Base": INFO[0006] Error while retrieving image from cache: getting file info: stat /cache/sha256:1bbdea4846231d91cce6c7ff3907d26fca444fd6b7e3c282b90c7fe4251f9f86: no such file or directory 
Step #0 - "Build Base": INFO[0006] Downloading base image ubuntu:18.04          
Step #0 - "Build Base": INFO[0007] Using files from context: [/workspace/cb/setup_env] 
Step #0 - "Build Base": INFO[0007] Using files from context: [/workspace/requirements.txt] 
Step #0 - "Build Base": INFO[0007] Checking for cached layer gcr.io/XXXXXXX/cloudbuild-helper-base/cache:38fcef1e4ae245952125c9ecc553979980a3becb2e89106e1652d9810674b746... 
Step #0 - "Build Base": INFO[0007] Using caching version of cmd: RUN /tmp/cb/setup_env && rm -rf /tmp/* 
Step #0 - "Build Base": INFO[0007] Skipping unpacking as no commands require it. 
Step #0 - "Build Base": INFO[0007] Taking snapshot of full filesystem...        
Step #0 - "Build Base": INFO[0008] ENV DEBIAN_FRONTEND=noninteractive           
Step #0 - "Build Base": INFO[0008] No files changed in this command, skipping snapshotting. 
Step #0 - "Build Base": INFO[0008] ENV APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1   
Step #0 - "Build Base": INFO[0008] No files changed in this command, skipping snapshotting. 
Step #0 - "Build Base": INFO[0008] ENV LANG en_US.UTF-8                         
Step #0 - "Build Base": INFO[0008] No files changed in this command, skipping snapshotting. 
Step #0 - "Build Base": INFO[0008] ENV LANGUAGE en_US:en                        
Step #0 - "Build Base": INFO[0008] No files changed in this command, skipping snapshotting. 
Step #0 - "Build Base": INFO[0008] ENV LC_ALL en_US.UTF-8                       
Step #0 - "Build Base": INFO[0008] No files changed in this command, skipping snapshotting. 
Step #0 - "Build Base": INFO[0008] ENV LC_CTYPE en_US.UTF-8                     
Step #0 - "Build Base": INFO[0008] No files changed in this command, skipping snapshotting. 
Step #0 - "Build Base": INFO[0008] COPY --from=gcr.io/kaniko-project/executor /kaniko / 
Step #0 - "Build Base": INFO[0008] Taking snapshot of files...                  
Step #0 - "Build Base": INFO[0009] Using files from context: [/workspace/cb/setup_env] 
Step #0 - "Build Base": INFO[0009] COPY cb/setup_env /tmp/cb/                   
Step #0 - "Build Base": INFO[0009] Taking snapshot of files...                  
Step #0 - "Build Base": INFO[0009] Using files from context: [/workspace/requirements.txt] 
Step #0 - "Build Base": INFO[0009] COPY requirements.txt /tmp/                  
Step #0 - "Build Base": INFO[0009] Taking snapshot of files...                  
Step #0 - "Build Base": INFO[0009] RUN /tmp/cb/setup_env && rm -rf /tmp/*       
Step #0 - "Build Base": INFO[0009] Found cached layer, extracting to filesystem 
Step #0 - "Build Base": INFO[0029] Taking snapshot of files...                  
Step #0 - "Build Base": fatal error: runtime: out of memory
Step #0 - "Build Base": 
Step #0 - "Build Base": runtime stack:
Step #0 - "Build Base": runtime.throw(0x1a1ec41, 0x16)
Step #0 - "Build Base":     /usr/local/go/src/runtime/panic.go:616 +0x81
Step #0 - "Build Base": runtime.sysMap(0xc4945b0000, 0x64050000, 0x7f3c02019100, 0x27e0f98)
Step #0 - "Build Base":     /usr/local/go/src/runtime/mem_linux.go:216 +0x20a
Step #0 - "Build Base": runtime.(*mheap).sysAlloc(0x27c7100, 0x64050000, 0x7f3c02019600)
Step #0 - "Build Base":     /usr/local/go/src/runtime/malloc.go:470 +0xd4
Step #0 - "Build Base": runtime.(*mheap).grow(0x27c7100, 0x32022, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/runtime/mheap.go:907 +0x60
Step #0 - "Build Base": runtime.(*mheap).allocSpanLocked(0x27c7100, 0x32022, 0x27e0fa8, 0xc420061ee0)
Step #0 - "Build Base":     /usr/local/go/src/runtime/mheap.go:820 +0x301
Step #0 - "Build Base": runtime.(*mheap).alloc_m(0x27c7100, 0x32022, 0xffffffffffff0101, 0xc420061f10)
Step #0 - "Build Base":     /usr/local/go/src/runtime/mheap.go:686 +0x118
Step #0 - "Build Base": runtime.(*mheap).alloc.func1()
Step #0 - "Build Base":     /usr/local/go/src/runtime/mheap.go:753 +0x4d
Step #0 - "Build Base": runtime.(*mheap).alloc(0x27c7100, 0x32022, 0xc420010101, 0x413ecc)
Step #0 - "Build Base":     /usr/local/go/src/runtime/mheap.go:752 +0x8a
Step #0 - "Build Base": runtime.largeAlloc(0x640431db, 0x450101, 0x7f3c02547000)
Step #0 - "Build Base":     /usr/local/go/src/runtime/malloc.go:826 +0x94
Step #0 - "Build Base": runtime.mallocgc.func1()
Step #0 - "Build Base":     /usr/local/go/src/runtime/malloc.go:721 +0x46
Step #0 - "Build Base": runtime.systemstack(0x0)
Step #0 - "Build Base":     /usr/local/go/src/runtime/asm_amd64.s:409 +0x79
Step #0 - "Build Base": runtime.mstart()
Step #0 - "Build Base":     /usr/local/go/src/runtime/proc.go:1175
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 1 [running]:
Step #0 - "Build Base": runtime.systemstack_switch()
Step #0 - "Build Base":     /usr/local/go/src/runtime/asm_amd64.s:363 fp=0xc420257258 sp=0xc420257250 pc=0x453b00
Step #0 - "Build Base": runtime.mallocgc(0x640431db, 0x1708040, 0xc420024901, 0xc420257330)
Step #0 - "Build Base":     /usr/local/go/src/runtime/malloc.go:720 +0x8a2 fp=0xc4202572f8 sp=0xc420257258 pc=0x4104a2
Step #0 - "Build Base": runtime.makeslice(0x1708040, 0x640431db, 0x640431db, 0x3e5b, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/runtime/slice.go:61 +0x77 fp=0xc420257328 sp=0xc4202572f8 pc=0x43fd17
Step #0 - "Build Base": bytes.makeSlice(0x640431db, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/bytes/buffer.go:230 +0x6d fp=0xc420257368 sp=0xc420257328 pc=0x4e437d
Step #0 - "Build Base": bytes.(*Buffer).grow(0xc4201a6150, 0x3e5b, 0xc420257428)
Step #0 - "Build Base":     /usr/local/go/src/bytes/buffer.go:144 +0x151 fp=0xc4202573b8 sp=0xc420257368 pc=0x4e3d31
Step #0 - "Build Base": bytes.(*Buffer).Write(0xc4201a6150, 0xc494502000, 0x3e5b, 0x8000, 0x0, 0xc494502000, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/bytes/buffer.go:174 +0xd9 fp=0xc4202573e8 sp=0xc4202573b8 pc=0x4e4009
Step #0 - "Build Base": archive/tar.(*regFileWriter).Write(0xc4944d63c0, 0xc494502000, 0x3e5b, 0x8000, 0x3e5b, 0x1ba28c0, 0xc420070040)
Step #0 - "Build Base":     /usr/local/go/src/archive/tar/writer.go:488 +0xe5 fp=0xc420257438 sp=0xc4202573e8 pc=0x8c7015
Step #0 - "Build Base": archive/tar.(*Writer).Write(0xc4215a3500, 0xc494502000, 0x3e5b, 0x8000, 0x3e5b, 0x1ba28c0, 0xc420070040)
Step #0 - "Build Base":     /usr/local/go/src/archive/tar/writer.go:426 +0x7c fp=0xc4202574a0 sp=0xc420257438 pc=0x8c6c9c
Step #0 - "Build Base": io.copyBuffer(0x1ba23a0, 0xc4215a3500, 0x1ba2380, 0xc42026efc0, 0xc494502000, 0x8000, 0x8000, 0x0, 0x0, 0xc4215a3528)
Step #0 - "Build Base":     /usr/local/go/src/io/io.go:402 +0x240 fp=0xc420257510 sp=0xc4202574a0 pc=0x460410
Step #0 - "Build Base": io.Copy(0x1ba23a0, 0xc4215a3500, 0x1ba2380, 0xc42026efc0, 0x2d35, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/io/io.go:362 +0x5a fp=0xc420257570 sp=0xc420257510 pc=0x46009a
Step #0 - "Build Base": github.com/GoogleContainerTools/kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/mutate.layerTime(0x1bd2640, 0xc4205680c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/mutate/mutate.go:512 +0x3d9 fp=0xc420257688 sp=0xc420257570 pc=0x1314949
Step #0 - "Build Base": github.com/GoogleContainerTools/kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/mutate.Time(0x1bdc900, 0xc420025b80, 0x0, 0x0, 0x0, 0x1bdc900, 0xc420025b80, 0x0, 0x0)
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/mutate/mutate.go:449 +0x1ac fp=0xc420257840 sp=0xc420257688 pc=0x1313f9c
Step #0 - "Build Base": github.com/GoogleContainerTools/kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/mutate.Canonical(0x1bdc900, 0xc420025b80, 0xbf5a7c1499299070, 0xcc53a8e03, 0x27c03a0, 0x1bdc900)
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/google/go-containerregistry/pkg/v1/mutate/mutate.go:545 +0x4a fp=0xc420257898 sp=0xc420257840 pc=0x1314e8a
Step #0 - "Build Base": github.com/GoogleContainerTools/kaniko/pkg/executor.DoBuild(0x27c0840, 0x1, 0x0, 0x0, 0x1c)
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/pkg/executor/build.go:463 +0x922 fp=0xc420257c28 sp=0xc420257898 pc=0x15cd2e2
Step #0 - "Build Base": github.com/GoogleContainerTools/kaniko/cmd/executor/cmd.glob..func2(0x27b0660, 0xc4205d5180, 0x2, 0x7)
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/cmd/executor/cmd/root.go:87 +0x9f fp=0xc420257d00 sp=0xc420257c28 pc=0x15e169f
Step #0 - "Build Base": github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).execute(0x27b0660, 0xc420038190, 0x7, 0x7, 0x27b0660, 0xc420038190)
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:766 +0x2c1 fp=0xc420257df0 sp=0xc420257d00 pc=0x15d7eb1
Step #0 - "Build Base": github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x27b0660, 0x0, 0x1a380be, 0x25)
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:852 +0x30a fp=0xc420257f30 sp=0xc420257df0 pc=0x15d8a6a
Step #0 - "Build Base": github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra.(*Command).Execute(0x27b0660, 0xc4205f9f78, 0xc42003e0b8)
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/spf13/cobra/command.go:800 +0x2b fp=0xc420257f60 sp=0xc420257f30 pc=0x15d873b
Step #0 - "Build Base": main.main()
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/cmd/executor/main.go:26 +0x2d fp=0xc420257f88 sp=0xc420257f60 pc=0x15e1bcd
Step #0 - "Build Base": runtime.main()
Step #0 - "Build Base":     /usr/local/go/src/runtime/proc.go:198 +0x212 fp=0xc420257fe0 sp=0xc420257f88 pc=0x42b7d2
Step #0 - "Build Base": runtime.goexit()
Step #0 - "Build Base":     /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420257fe8 sp=0xc420257fe0 pc=0x456511
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 6 [select]:
Step #0 - "Build Base": github.com/GoogleContainerTools/kaniko/vendor/go.opencensus.io/stats/view.(*worker).start(0xc420072200)
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/vendor/go.opencensus.io/stats/view/worker.go:146 +0x11d
Step #0 - "Build Base": created by github.com/GoogleContainerTools/kaniko/vendor/go.opencensus.io/stats/view.init.0
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/vendor/go.opencensus.io/stats/view/worker.go:29 +0x58
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 8 [chan receive]:
Step #0 - "Build Base": github.com/GoogleContainerTools/kaniko/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x27c0680)
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/golang/glog/glog.go:882 +0x8b
Step #0 - "Build Base": created by github.com/GoogleContainerTools/kaniko/vendor/github.com/golang/glog.init.0
Step #0 - "Build Base":     /go/src/github.com/GoogleContainerTools/kaniko/vendor/github.com/golang/glog/glog.go:410 +0x203
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 19 [syscall, 1 minutes]:
Step #0 - "Build Base": os/signal.signal_recv(0x0)
Step #0 - "Build Base":     /usr/local/go/src/runtime/sigqueue.go:139 +0xa6
Step #0 - "Build Base": os/signal.loop()
Step #0 - "Build Base":     /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
Step #0 - "Build Base": created by os/signal.init.0
Step #0 - "Build Base":     /usr/local/go/src/os/signal/signal_unix.go:28 +0x41
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 140 [select]:
Step #0 - "Build Base": net/http.(*persistConn).writeLoop(0xc420205440)
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1822 +0x14b
Step #0 - "Build Base": created by net/http.(*Transport).dialConn
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1238 +0x97f
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 16 [IO wait, 1 minutes]:
Step #0 - "Build Base": internal/poll.runtime_pollWait(0x7f3c024ebf00, 0x72, 0xc42019d9a8)
Step #0 - "Build Base":     /usr/local/go/src/runtime/netpoll.go:173 +0x57
Step #0 - "Build Base": internal/poll.(*pollDesc).wait(0xc4203b8498, 0x72, 0xffffffffffffff00, 0x1ba7880, 0x273c638)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
Step #0 - "Build Base": internal/poll.(*pollDesc).waitRead(0xc4203b8498, 0xc420379000, 0x1000, 0x1000)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Step #0 - "Build Base": internal/poll.(*FD).Read(0xc4203b8480, 0xc420379000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
Step #0 - "Build Base": net.(*netFD).Read(0xc4203b8480, 0xc420379000, 0x1000, 0x1000, 0x452580, 0xc420000180, 0x4)
Step #0 - "Build Base":     /usr/local/go/src/net/fd_unix.go:202 +0x4f
Step #0 - "Build Base": net.(*conn).Read(0xc42000c040, 0xc420379000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/net/net.go:176 +0x6a
Step #0 - "Build Base": net/http.(*persistConn).Read(0xc4203bf0e0, 0xc420379000, 0x1000, 0x1000, 0xc42019db98, 0x403355, 0xc42003e540)
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1453 +0x136
Step #0 - "Build Base": bufio.(*Reader).fill(0xc42047a720)
Step #0 - "Build Base":     /usr/local/go/src/bufio/bufio.go:100 +0x11e
Step #0 - "Build Base": bufio.(*Reader).Peek(0xc42047a720, 0x1, 0x0, 0x0, 0x0, 0xc42003e660, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/bufio/bufio.go:132 +0x3a
Step #0 - "Build Base": net/http.(*persistConn).readLoop(0xc4203bf0e0)
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1601 +0x185
Step #0 - "Build Base": created by net/http.(*Transport).dialConn
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1237 +0x95a
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 17 [select, 1 minutes]:
Step #0 - "Build Base": net/http.(*persistConn).writeLoop(0xc4203bf0e0)
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1822 +0x14b
Step #0 - "Build Base": created by net/http.(*Transport).dialConn
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1238 +0x97f
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 139 [IO wait]:
Step #0 - "Build Base": internal/poll.runtime_pollWait(0x7f3c024ebbc0, 0x72, 0xc42019f760)
Step #0 - "Build Base":     /usr/local/go/src/runtime/netpoll.go:173 +0x57
Step #0 - "Build Base": internal/poll.(*pollDesc).wait(0xc420038b18, 0x72, 0xffffffffffffff00, 0x1ba7880, 0x273c638)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
Step #0 - "Build Base": internal/poll.(*pollDesc).waitRead(0xc420038b18, 0xc420f2c800, 0x800, 0x800)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Step #0 - "Build Base": internal/poll.(*FD).Read(0xc420038b00, 0xc420f2c800, 0x800, 0x800, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
Step #0 - "Build Base": net.(*netFD).Read(0xc420038b00, 0xc420f2c800, 0x800, 0x800, 0x0, 0xc420665350, 0x1)
Step #0 - "Build Base":     /usr/local/go/src/net/fd_unix.go:202 +0x4f
Step #0 - "Build Base": net.(*conn).Read(0xc42000cbe0, 0xc420f2c800, 0x800, 0x800, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/net/net.go:176 +0x6a
Step #0 - "Build Base": crypto/tls.(*block).readFromUntil(0xc4206fe750, 0x7f3c024fdc70, 0xc42000cbe0, 0x5, 0xc42000cbe0, 0x6)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:493 +0x96
Step #0 - "Build Base": crypto/tls.(*Conn).readRecord(0xc420ca2000, 0x1ab9517, 0xc420ca2120, 0xc42019fe58)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:595 +0xe0
Step #0 - "Build Base": crypto/tls.(*Conn).Read(0xc420ca2000, 0xc42001a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:1156 +0x100
Step #0 - "Build Base": net/http.(*persistConn).Read(0xc420205440, 0xc42001a000, 0x1000, 0x1000, 0x27af5c0, 0xc420205440, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1453 +0x136
Step #0 - "Build Base": bufio.(*Reader).fill(0xc4205697a0)
Step #0 - "Build Base":     /usr/local/go/src/bufio/bufio.go:100 +0x11e
Step #0 - "Build Base": bufio.(*Reader).Peek(0xc4205697a0, 0x1, 0x0, 0x0, 0x0, 0xc421888ea0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/bufio/bufio.go:132 +0x3a
Step #0 - "Build Base": net/http.(*persistConn).readLoop(0xc420205440)
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1601 +0x185
Step #0 - "Build Base": created by net/http.(*Transport).dialConn
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1237 +0x95a
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 98 [IO wait, 1 minutes]:
Step #0 - "Build Base": internal/poll.runtime_pollWait(0x7f3c024ebd60, 0x72, 0xc42019c760)
Step #0 - "Build Base":     /usr/local/go/src/runtime/netpoll.go:173 +0x57
Step #0 - "Build Base": internal/poll.(*pollDesc).wait(0xc420039718, 0x72, 0xffffffffffffff00, 0x1ba7880, 0x273c638)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
Step #0 - "Build Base": internal/poll.(*pollDesc).waitRead(0xc420039718, 0xc420ef0000, 0x2000, 0x2000)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Step #0 - "Build Base": internal/poll.(*FD).Read(0xc420039700, 0xc420ef0000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
Step #0 - "Build Base": net.(*netFD).Read(0xc420039700, 0xc420ef0000, 0x2000, 0x2000, 0x20, 0x28, 0xc420d45a40)
Step #0 - "Build Base":     /usr/local/go/src/net/fd_unix.go:202 +0x4f
Step #0 - "Build Base": net.(*conn).Read(0xc42000cb68, 0xc420ef0000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/net/net.go:176 +0x6a
Step #0 - "Build Base": crypto/tls.(*block).readFromUntil(0xc420ea9da0, 0x7f3c024fdc70, 0xc42000cb68, 0x5, 0xc42000cb68, 0x5)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:493 +0x96
Step #0 - "Build Base": crypto/tls.(*Conn).readRecord(0xc420ca3c00, 0x1ab9517, 0xc420ca3d20, 0x452580)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:595 +0xe0
Step #0 - "Build Base": crypto/tls.(*Conn).Read(0xc420ca3c00, 0xc420ec5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:1156 +0x100
Step #0 - "Build Base": net/http.(*persistConn).Read(0xc4201c2c60, 0xc420ec5000, 0x1000, 0x1000, 0xc42019cb98, 0x403355, 0xc42003e660)
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1453 +0x136
Step #0 - "Build Base": bufio.(*Reader).fill(0xc420569080)
Step #0 - "Build Base":     /usr/local/go/src/bufio/bufio.go:100 +0x11e
Step #0 - "Build Base": bufio.(*Reader).Peek(0xc420569080, 0x1, 0x0, 0x0, 0x0, 0xc420713200, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/bufio/bufio.go:132 +0x3a
Step #0 - "Build Base": net/http.(*persistConn).readLoop(0xc4201c2c60)
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1601 +0x185
Step #0 - "Build Base": created by net/http.(*Transport).dialConn
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1237 +0x95a
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 32 [IO wait]:
Step #0 - "Build Base": internal/poll.runtime_pollWait(0x7f3c024ebe30, 0x72, 0xc42019a858)
Step #0 - "Build Base":     /usr/local/go/src/runtime/netpoll.go:173 +0x57
Step #0 - "Build Base": internal/poll.(*pollDesc).wait(0xc420038b98, 0x72, 0xffffffffffffff00, 0x1ba7880, 0x273c638)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
Step #0 - "Build Base": internal/poll.(*pollDesc).waitRead(0xc420038b98, 0xc420515000, 0x1000, 0x1000)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Step #0 - "Build Base": internal/poll.(*FD).Read(0xc420038b80, 0xc420515000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
Step #0 - "Build Base": net.(*netFD).Read(0xc420038b80, 0xc420515000, 0x1000, 0x1000, 0x432b59, 0xc42037ed80, 0x200000003)
Step #0 - "Build Base":     /usr/local/go/src/net/fd_unix.go:202 +0x4f
Step #0 - "Build Base": net.(*conn).Read(0xc42000c680, 0xc420515000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/net/net.go:176 +0x6a
Step #0 - "Build Base": crypto/tls.(*block).readFromUntil(0xc420597980, 0x7f3c024fdc70, 0xc42000c680, 0x5, 0xc42000c680, 0xc42019aa10)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:493 +0x96
Step #0 - "Build Base": crypto/tls.(*Conn).readRecord(0xc4205a2e00, 0x1ab9517, 0xc4205a2f20, 0xc42037ed80)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:595 +0xe0
Step #0 - "Build Base": crypto/tls.(*Conn).Read(0xc4205a2e00, 0xc4203ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:1156 +0x100
Step #0 - "Build Base": bufio.(*Reader).Read(0xc42047a600, 0xc4202ac118, 0x9, 0x9, 0xc4205a3130, 0x5fd340, 0xc4205a3130)
Step #0 - "Build Base":     /usr/local/go/src/bufio/bufio.go:216 +0x238
Step #0 - "Build Base": io.ReadAtLeast(0x1ba2440, 0xc42047a600, 0xc4202ac118, 0x9, 0x9, 0x9, 0x67a151, 0xc4205a2e00, 0xc42018b000)
Step #0 - "Build Base":     /usr/local/go/src/io/io.go:309 +0x86
Step #0 - "Build Base": io.ReadFull(0x1ba2440, 0xc42047a600, 0xc4202ac118, 0x9, 0x9, 0x6c0bb4, 0x7f3c02506db0, 0xc4205a2e00)
Step #0 - "Build Base":     /usr/local/go/src/io/io.go:327 +0x58
Step #0 - "Build Base": net/http.http2readFrameHeader(0xc4202ac118, 0x9, 0x9, 0x1ba2440, 0xc42047a600, 0x0, 0xc400000000, 0x429374, 0x1ab9598)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:1517 +0x7b
Step #0 - "Build Base": net/http.(*http2Framer).ReadFrame(0xc4202ac0e0, 0xc4206ea560, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:1775 +0xa4
Step #0 - "Build Base": net/http.(*http2clientConnReadLoop).run(0xc42019afb8, 0x1ab8ed0, 0xc420055fb8)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:8028 +0x90
Step #0 - "Build Base": net/http.(*http2ClientConn).readLoop(0xc4205101c0)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:7956 +0x68
Step #0 - "Build Base": created by net/http.(*http2Transport).newClientConn
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:7143 +0x65c
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 54 [IO wait]:
Step #0 - "Build Base": internal/poll.runtime_pollWait(0x7f3c024ebaf0, 0x72, 0xc42019b858)
Step #0 - "Build Base":     /usr/local/go/src/runtime/netpoll.go:173 +0x57
Step #0 - "Build Base": internal/poll.(*pollDesc).wait(0xc420038518, 0x72, 0xffffffffffffff00, 0x1ba7880, 0x273c638)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
Step #0 - "Build Base": internal/poll.(*pollDesc).waitRead(0xc420038518, 0xc4208b6000, 0x4000, 0x4000)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Step #0 - "Build Base": internal/poll.(*FD).Read(0xc420038500, 0xc4208b6000, 0x4000, 0x4000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
Step #0 - "Build Base": net.(*netFD).Read(0xc420038500, 0xc4208b6000, 0x4000, 0x4000, 0x432b59, 0xc42037fe00, 0x200000003)
Step #0 - "Build Base":     /usr/local/go/src/net/fd_unix.go:202 +0x4f
Step #0 - "Build Base": net.(*conn).Read(0xc42000ca88, 0xc4208b6000, 0x4000, 0x4000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/net/net.go:176 +0x6a
Step #0 - "Build Base": crypto/tls.(*block).readFromUntil(0xc420358d80, 0x7f3c024fdc70, 0xc42000ca88, 0x5, 0xc42000ca88, 0xc42019ba10)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:493 +0x96
Step #0 - "Build Base": crypto/tls.(*Conn).readRecord(0xc4205a3880, 0x1ab9517, 0xc4205a39a0, 0xc42037fe00)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:595 +0xe0
Step #0 - "Build Base": crypto/tls.(*Conn).Read(0xc4205a3880, 0xc4204f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:1156 +0x100
Step #0 - "Build Base": bufio.(*Reader).Read(0xc420568c60, 0xc4202ac738, 0x9, 0x9, 0xc4205a3bb0, 0x5fd340, 0xc4205a3bb0)
Step #0 - "Build Base":     /usr/local/go/src/bufio/bufio.go:216 +0x238
Step #0 - "Build Base": io.ReadAtLeast(0x1ba2440, 0xc420568c60, 0xc4202ac738, 0x9, 0x9, 0x9, 0x67a151, 0xc4205a3880, 0xc4204f8000)
Step #0 - "Build Base":     /usr/local/go/src/io/io.go:309 +0x86
Step #0 - "Build Base": io.ReadFull(0x1ba2440, 0xc420568c60, 0xc4202ac738, 0x9, 0x9, 0x6c0bb4, 0x7f3c02506db0, 0xc4205a3880)
Step #0 - "Build Base":     /usr/local/go/src/io/io.go:327 +0x58
Step #0 - "Build Base": net/http.http2readFrameHeader(0xc4202ac738, 0x9, 0x9, 0x1ba2440, 0xc420568c60, 0x0, 0xc400000000, 0x429374, 0x1ab9598)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:1517 +0x7b
Step #0 - "Build Base": net/http.(*http2Framer).ReadFrame(0xc4202ac700, 0xc420c868e0, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:1775 +0xa4
Step #0 - "Build Base": net/http.(*http2clientConnReadLoop).run(0xc42019bfb8, 0x1ab8ed0, 0xc42004efb8)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:8028 +0x90
Step #0 - "Build Base": net/http.(*http2ClientConn).readLoop(0xc420193340)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:7956 +0x68
Step #0 - "Build Base": created by net/http.(*http2Transport).newClientConn
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:7143 +0x65c
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 99 [select, 1 minutes]:
Step #0 - "Build Base": net/http.(*persistConn).writeLoop(0xc4201c2c60)
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1822 +0x14b
Step #0 - "Build Base": created by net/http.(*Transport).dialConn
Step #0 - "Build Base":     /usr/local/go/src/net/http/transport.go:1238 +0x97f
Step #0 - "Build Base": 
Step #0 - "Build Base": goroutine 123 [IO wait]:
Step #0 - "Build Base": internal/poll.runtime_pollWait(0x7f3c024eba20, 0x72, 0xc420e90858)
Step #0 - "Build Base":     /usr/local/go/src/runtime/netpoll.go:173 +0x57
Step #0 - "Build Base": internal/poll.(*pollDesc).wait(0xc420f09898, 0x72, 0xffffffffffffff00, 0x1ba7880, 0x273c638)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
Step #0 - "Build Base": internal/poll.(*pollDesc).waitRead(0xc420f09898, 0xc422278000, 0x8000, 0x8000)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Step #0 - "Build Base": internal/poll.(*FD).Read(0xc420f09880, 0xc422278000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
Step #0 - "Build Base": net.(*netFD).Read(0xc420f09880, 0xc422278000, 0x8000, 0x8000, 0x8, 0x8, 0x7ff3)
Step #0 - "Build Base":     /usr/local/go/src/net/fd_unix.go:202 +0x4f
Step #0 - "Build Base": net.(*conn).Read(0xc42000ce00, 0xc422278000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/net/net.go:176 +0x6a
Step #0 - "Build Base": crypto/tls.(*block).readFromUntil(0xc420f376b0, 0x7f3c024fdc70, 0xc42000ce00, 0x5, 0xc42000ce00, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:493 +0x96
Step #0 - "Build Base": crypto/tls.(*Conn).readRecord(0xc420ecf500, 0x1ab9517, 0xc420ecf620, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:595 +0xe0
Step #0 - "Build Base": crypto/tls.(*Conn).Read(0xc420ecf500, 0xc420519000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/crypto/tls/conn.go:1156 +0x100
Step #0 - "Build Base": bufio.(*Reader).Read(0xc42047a660, 0xc4206ca038, 0x9, 0x9, 0xc420e90c50, 0x45c983, 0xc420e90c60)
Step #0 - "Build Base":     /usr/local/go/src/bufio/bufio.go:216 +0x238
Step #0 - "Build Base": io.ReadAtLeast(0x1ba2440, 0xc42047a660, 0xc4206ca038, 0x9, 0x9, 0x9, 0xc420522b01, 0xc420e90d00, 0x68444b)
Step #0 - "Build Base":     /usr/local/go/src/io/io.go:309 +0x86
Step #0 - "Build Base": io.ReadFull(0x1ba2440, 0xc42047a660, 0xc4206ca038, 0x9, 0x9, 0xc420070040, 0xc420e90d30, 0x68435d)
Step #0 - "Build Base":     /usr/local/go/src/io/io.go:327 +0x58
Step #0 - "Build Base": net/http.http2readFrameHeader(0xc4206ca038, 0x9, 0x9, 0x1ba2440, 0xc42047a660, 0x0, 0x0, 0xc4202a25d0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:1517 +0x7b
Step #0 - "Build Base": net/http.(*http2Framer).ReadFrame(0xc4206ca000, 0xc4202a25d0, 0x0, 0x0, 0x0)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:1775 +0xa4
Step #0 - "Build Base": net/http.(*http2clientConnReadLoop).run(0xc420e90fb8, 0x1ab8ed0, 0xc42004efb8)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:8028 +0x90
Step #0 - "Build Base": net/http.(*http2ClientConn).readLoop(0xc420637340)
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:7956 +0x68
Step #0 - "Build Base": created by net/http.(*http2Transport).newClientConn
Step #0 - "Build Base":     /usr/local/go/src/net/http/h2_bundle.go:7143 +0x65c
Finished Step #0 - "Build Base"
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor" failed: exit status 2
tejal29 commented 5 years ago

@ptinsley This is not what we expect. Can you confirm if when the build crashed there were any other GCB builds running in parallel?

/cc @ImJasonH would you happen to know why this is happening?

ptinsley commented 5 years ago

No parallel steps in this specific build.

imjasonh commented 5 years ago

Does this error go away if you request a build with a larger --machine-type? My guess is that the contents of the source context are just larger than the 3.75 GB of memory (minus GCB's own overhead) available to the build, and so you're getting OOMs.

Kaniko could buffer its layer contents to disk, then upload to the registry from there, or it could stream contents to the registry instead of buffering in memory. (If it does either of those already then I don't know why we'd be getting an OOM)

tejal29 commented 5 years ago

Thanks @ImJasonH.

ptinsley commented 5 years ago

@ImJasonH I'll try that on Monday but it would seem strange for that to be the cause and it works fine when the build is retried...

FWIW the finished build when it works shows as 977MB. The base that it comes from is 218MB.

donmccasland commented 4 years ago

Any update on this?

ptinsley commented 4 years ago

I had a lot of other stuff jump in front of this, I changed the image size a couple of days ago and am monitoring for failures, we don't build this specific repo a ton so it will take a few days to know for sure

agrinh commented 4 years ago

+1, I've run into this as well

firaxis commented 4 years ago

Running into same issue. Can I help with any more details ?

pkutishch commented 3 years ago

+1 here, running thru this issue as well