search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
8.28k stars 1.17k forks source link

Stack overflow during cache export #1609

Closed myhau closed 3 years ago

myhau commented 4 years ago

Environment

Daemon

Client

What happens

When building multi-stage buildkit daemon fails and exits with:

runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow

(full logs below)

Details

Used Dockerfile

FROM node:11.6-slim as base-build

# build args
ARG SOME_ARG
ARG HOME=/home/app
ARG NPMRC_PATH=.npmrc

ENV NODE_ENV=development

# setup unprivileged user
RUN rm -rf $HOME && useradd -Umrd $HOME app
USER app
WORKDIR $HOME

FROM base-build as node-base-build

COPY --chown=app:app "bin" "bin"
COPY --chown=app:app "package.json" "package-lock.json" ./

RUN $HOME/bin/write_npmrc \
    && npm ci --production=false

COPY --chown=app:app "webpack.config.js" ".babelrc" ./
COPY --chown=app:app "config" "config"

RUN mv "config/default.local.json" "config/default.json"

FROM node-base-build as lint-and-test

USER app

COPY --chown=app:app "src" "src"
COPY --chown=app:app "test" "test"
COPY --chown=app:app "jest.config.js" "mocha.opts" "mocha-globals.js" "tsconfig.json" ".eslintrc" ".eslintignore" ".npmignore" ./

RUN npm test && \
    npm run check-node-version && \
    ./node_modules/.bin/eslint --env-info && \
    npm run lint

FROM lint-and-test as prod-build

ENV NODE_ENV=production
RUN npm run build && \
    npm prune --production && \
    rm -rf $HOME/src && \
    rm -rf $HOME/dist/config && \
    mv $HOME/dist/* .

CMD ["npm", "start"]

Used buildctl command

buildctl \
    --debug \
    --addr tcp://127.0.0.1:1234 \
    --tlscacert ca.pem \
    --tlscert cert.pem \
    --tlskey key.pem \
    build \
    --frontend=dockerfile.v0 \
    --local context=path \
    --local dockerfile=path \
    --opt filename=Dockerfile.azure \
    --output type=image,name=someregistry.azurecr.io/app:v1,push=true \
    --export-cache type=registry,ref="someregistry.azurecr.io/app:cache,mode=max" \
    --import-cache type=registry,ref="someregistry.azurecr.io/app:cache" \
    --opt build-arg:SOME_ARG=VAL \
    --opt target=prod-build

Last buildctl logs

... REDACTED ...
#20 exporting to image
#20 exporting layers
#20 exporting layers 13.1s done
#20 exporting manifest sha256:bc5ea0c60d9b877e875d5b76c9c3ba1b80906c03e864d979174c76cc8cfa029a 0.0s done
#20 exporting config sha256:fc447ab50d4f8a91cb532ce78930e1817e7d38fff9d88f86aef33482e60ec228 0.0s done
#20 pushing layers
#20 pushing layers 1.0s done
#20 pushing manifest for someregistry.azurecr.io/app:198077c4d84155ac78712bdd37cce477b8cdd018
#20 pushing manifest for someregistry.azurecr.io/app:198077c4d84155ac78712bdd37cce477b8cdd018 0.7s done
#20 DONE 15.0s

#21 exporting cache
#21 preparing build cache for export
#21 preparing build cache for export 1.0s done
DEBU[0201] stopping session
Handling connection for 1234
error: rpc error: code = Unavailable desc = transport is closing
failed to receive status
github.com/moby/buildkit/client.(*Client).solve.func4
    /src/client/solve.go:258
golang.org/x/sync/errgroup.(*Group).Go.func1
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
runtime.goexit

Full daemon logs

time="2020-07-28T15:37:15Z" level=info msg="auto snapshotter: using overlayfs"
time="2020-07-28T15:37:15Z" level=warning msg="NoProcessSandbox is enabled. Note that NoProcessSandbox allows build containers to kill (and potentially ptrace) an arbitrary process in the BuildKit host namespace. NoProcessSandbox should be enabled only when the BuildKit is running in a container as an unprivileged user."
time="2020-07-28T15:37:15Z" level=info msg="found worker \"ofsekn4bi0peuur6tuqp2fulx\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:buildkitd-5546bcff4b-zcggq org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/386]"
time="2020-07-28T15:37:15Z" level=warning msg="rootless mode is not supported for containerd workers. disabling containerd worker."
time="2020-07-28T15:37:15Z" level=info msg="found 1 workers, default=\"ofsekn4bi0peuur6tuqp2fulx\""
time="2020-07-28T15:37:15Z" level=warning msg="currently, only the default worker can be used."
time="2020-07-28T15:37:15Z" level=warning msg="TLS is disabled for unix:///run/user/1000/buildkit/buildkitd.sock"
time="2020-07-28T15:37:15Z" level=info msg="running server on /run/user/1000/buildkit/buildkitd.sock"
time="2020-07-28T15:37:15Z" level=info msg="running server on [::]:1234"
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow

runtime stack:
runtime.throw(0x11feb6d, 0xe)
    /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.newstack()
    /usr/local/go/src/runtime/stack.go:1047 +0x6e9
runtime.morestack()
    /usr/local/go/src/runtime/asm_amd64.s:449 +0x8f

goroutine 83 [running]:
runtime.heapBitsSetType(0xc00578d800, 0x30, 0x30, 0xff8260)
    /usr/local/go/src/runtime/mbitmap.go:938 +0xa4e fp=0xc023a063a0 sp=0xc023a06398 pc=0x4179de
runtime.mallocgc(0x30, 0xff8260, 0xa4c48c01, 0x33305920ff6aa9a4)
    /usr/local/go/src/runtime/malloc.go:1052 +0x53e fp=0xc023a06440 sp=0xc023a063a0 pc=0x40d8fe
runtime.makeslice(0xff8260, 0x2, 0x2, 0x1b78a40)
    /usr/local/go/src/runtime/slice.go:49 +0x6c fp=0xc023a06470 sp=0xc023a06440 pc=0x446dec
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88910, 0xc043a04910, 0xc023a066e0, 0xc00578d7d0)
    /src/cache/remotecache/v1/utils.go:277 +0x96 fp=0xc023a065e0 sp=0xc023a06470 pc=0xba1b36
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e888c0, 0xc043a04910, 0xc023a06850, 0xc00578bbc0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a06750 sp=0xc023a065e0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88870, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a068c0 sp=0xc023a06750 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88780, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a06a30 sp=0xc023a068c0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88af0, 0xc043a04910, 0xc023a06ca0, 0xc00578bb40)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a06ba0 sp=0xc023a06a30 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88aa0, 0xc043a04910, 0xc023a06e10, 0xc00578d740)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a06d10 sp=0xc023a06ba0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a50, 0xc043a04910, 0xc023a06f80, 0xc00578d710)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a06e80 sp=0xc023a06d10 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a00, 0xc043a04910, 0xc023a070f0, 0xc00578bb20)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a06ff0 sp=0xc023a06e80 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e889b0, 0xc043a04910, 0xc023a07260, 0xc00578d6e0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a07160 sp=0xc023a06ff0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88960, 0xc043a04910, 0xc023a073d0, 0xc00578d6b0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a072d0 sp=0xc023a07160 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88910, 0xc043a04910, 0xc023a07540, 0xc00578d680)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a07440 sp=0xc023a072d0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e888c0, 0xc043a04910, 0xc023a076b0, 0xc00578bb00)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a075b0 sp=0xc023a07440 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88870, 0xc043a04910, 0xc023a07820, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a07720 sp=0xc023a075b0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88780, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a07890 sp=0xc023a07720 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88af0, 0xc043a04910, 0xc023a07b00, 0xc00578baa0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a07a00 sp=0xc023a07890 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88aa0, 0xc043a04910, 0xc023a07c70, 0xc00578d5f0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a07b70 sp=0xc023a07a00 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a50, 0xc043a04910, 0xc023a07de0, 0xc00578d5c0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a07ce0 sp=0xc023a07b70 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a00, 0xc043a04910, 0xc023a07f50, 0xc00578ba80)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a07e50 sp=0xc023a07ce0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e889b0, 0xc043a04910, 0xc023a080c0, 0xc00578d590)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a07fc0 sp=0xc023a07e50 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88960, 0xc043a04910, 0xc023a08230, 0xc00578d560)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a08130 sp=0xc023a07fc0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88910, 0xc043a04910, 0xc023a083a0, 0xc00578d530)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a082a0 sp=0xc023a08130 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e888c0, 0xc043a04910, 0xc023a08510, 0xc00578ba60)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a08410 sp=0xc023a082a0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88870, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a08580 sp=0xc023a08410 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88780, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a086f0 sp=0xc023a08580 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88af0, 0xc043a04910, 0xc023a08960, 0xc00578b9e0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a08860 sp=0xc023a086f0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88aa0, 0xc043a04910, 0xc023a08ad0, 0xc00578d4a0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a089d0 sp=0xc023a08860 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a50, 0xc043a04910, 0xc023a08c40, 0xc00578d470)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a08b40 sp=0xc023a089d0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a00, 0xc043a04910, 0xc023a08db0, 0xc00578b9c0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a08cb0 sp=0xc023a08b40 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e889b0, 0xc043a04910, 0xc023a08f20, 0xc00578d440)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a08e20 sp=0xc023a08cb0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88960, 0xc043a04910, 0xc023a09090, 0xc00578d410)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a08f90 sp=0xc023a08e20 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88910, 0xc043a04910, 0xc023a09200, 0xc00578d3e0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a09100 sp=0xc023a08f90 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e888c0, 0xc043a04910, 0xc023a09370, 0xc00578b9a0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a09270 sp=0xc023a09100 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88870, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a093e0 sp=0xc023a09270 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88780, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a09550 sp=0xc023a093e0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88af0, 0xc043a04910, 0xc023a097c0, 0xc00578b920)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a096c0 sp=0xc023a09550 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88aa0, 0xc043a04910, 0xc023a09930, 0xc00578d350)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a09830 sp=0xc023a096c0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a50, 0xc043a04910, 0xc023a09aa0, 0xc00578d320)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a099a0 sp=0xc023a09830 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a00, 0xc043a04910, 0xc023a09c10, 0xc00578b900)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a09b10 sp=0xc023a099a0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e889b0, 0xc043a04910, 0xc023a09d80, 0xc00578d2f0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a09c80 sp=0xc023a09b10 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88960, 0xc043a04910, 0xc023a09ef0, 0xc00578d2c0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a09df0 sp=0xc023a09c80 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88910, 0xc043a04910, 0xc023a0a060, 0xc00578d290)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a09f60 sp=0xc023a09df0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e888c0, 0xc043a04910, 0xc023a0a1d0, 0xc00578b8e0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0a0d0 sp=0xc023a09f60 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88870, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0a240 sp=0xc023a0a0d0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88780, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0a3b0 sp=0xc023a0a240 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88c80, 0xc043a04910, 0xc023a0a620, 0xc00578b860)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0a520 sp=0xc023a0a3b0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88aa0, 0xc043a04910, 0xc023a0a790, 0xc00578d200)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0a690 sp=0xc023a0a520 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a50, 0xc043a04910, 0xc023a0a900, 0xc00578d1d0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0a800 sp=0xc023a0a690 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a00, 0xc043a04910, 0xc023a0aa70, 0xc00578b840)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0a970 sp=0xc023a0a800 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e889b0, 0xc043a04910, 0xc023a0abe0, 0xc00578d1a0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0aae0 sp=0xc023a0a970 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88960, 0xc043a04910, 0xc023a0ad50, 0xc00578d170)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0ac50 sp=0xc023a0aae0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88910, 0xc043a04910, 0xc023a0aec0, 0xc00578d140)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0adc0 sp=0xc023a0ac50 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e888c0, 0xc043a04910, 0xc023a0b030, 0xc00578b820)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0af30 sp=0xc023a0adc0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88870, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0b0a0 sp=0xc023a0af30 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88780, 0xc043a04910, 0xc023a0b310, 0xc00578d0e0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0b210 sp=0xc023a0b0a0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88af0, 0xc043a04910, 0xc023a0b480, 0xc00578b7c0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0b380 sp=0xc023a0b210 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88aa0, 0xc043a04910, 0xc023a0b5f0, 0xc00578d0b0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0b4f0 sp=0xc023a0b380 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a50, 0xc043a04910, 0xc023a0b760, 0xc00578d080)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0b660 sp=0xc023a0b4f0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a00, 0xc043a04910, 0xc023a0b8d0, 0xc00578b7a0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0b7d0 sp=0xc023a0b660 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e889b0, 0xc043a04910, 0xc023a0ba40, 0xc00578d050)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0b940 sp=0xc023a0b7d0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88960, 0xc043a04910, 0xc023a0bbb0, 0xc00578d020)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0bab0 sp=0xc023a0b940 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88910, 0xc043a04910, 0xc023a0bd20, 0xc00578cff0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0bc20 sp=0xc023a0bab0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e888c0, 0xc043a04910, 0xc023a0be90, 0xc00578b780)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0bd90 sp=0xc023a0bc20 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88870, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0bf00 sp=0xc023a0bd90 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88780, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0c070 sp=0xc023a0bf00 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88af0, 0xc043a04910, 0xc023a0c2e0, 0xc00578b700)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0c1e0 sp=0xc023a0c070 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88aa0, 0xc043a04910, 0xc023a0c450, 0xc00578cf60)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0c350 sp=0xc023a0c1e0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a50, 0xc043a04910, 0xc023a0c5c0, 0xc00578cf30)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0c4c0 sp=0xc023a0c350 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a00, 0xc043a04910, 0xc023a0c730, 0xc00578b6e0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0c630 sp=0xc023a0c4c0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e889b0, 0xc043a04910, 0xc023a0c8a0, 0xc00578cf00)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0c7a0 sp=0xc023a0c630 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88960, 0xc043a04910, 0xc023a0ca10, 0xc00578ced0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0c910 sp=0xc023a0c7a0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88910, 0xc043a04910, 0xc023a0cb80, 0xc00578cea0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0ca80 sp=0xc023a0c910 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e888c0, 0xc043a04910, 0xc023a0ccf0, 0xc00578b6c0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0cbf0 sp=0xc023a0ca80 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88870, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0cd60 sp=0xc023a0cbf0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88780, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0ced0 sp=0xc023a0cd60 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88af0, 0xc043a04910, 0xc023a0d140, 0xc00578b640)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0d040 sp=0xc023a0ced0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88aa0, 0xc043a04910, 0xc023a0d2b0, 0xc00578ce10)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0d1b0 sp=0xc023a0d040 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a50, 0xc043a04910, 0xc023a0d420, 0xc00578cde0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0d320 sp=0xc023a0d1b0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a00, 0xc043a04910, 0xc023a0d590, 0xc00578b620)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0d490 sp=0xc023a0d320 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e889b0, 0xc043a04910, 0xc023a0d700, 0xc00578cdb0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0d600 sp=0xc023a0d490 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88960, 0xc043a04910, 0xc023a0d870, 0xc00578cd80)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0d770 sp=0xc023a0d600 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88910, 0xc043a04910, 0xc023a0d9e0, 0xc00578cd50)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0d8e0 sp=0xc023a0d770 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e888c0, 0xc043a04910, 0xc023a0db50, 0xc00578b600)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0da50 sp=0xc023a0d8e0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88870, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0dbc0 sp=0xc023a0da50 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88780, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0dd30 sp=0xc023a0dbc0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88af0, 0xc043a04910, 0xc023a0dfa0, 0xc00578b580)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0dea0 sp=0xc023a0dd30 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88aa0, 0xc043a04910, 0xc023a0e110, 0xc00578ccc0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0e010 sp=0xc023a0dea0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a50, 0xc043a04910, 0xc023a0e280, 0xc00578cc90)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0e180 sp=0xc023a0e010 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a00, 0xc043a04910, 0xc023a0e3f0, 0xc00578b560)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0e2f0 sp=0xc023a0e180 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e889b0, 0xc043a04910, 0xc023a0e560, 0xc00578cc60)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0e460 sp=0xc023a0e2f0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88960, 0xc043a04910, 0xc023a0e6d0, 0xc00578cc30)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0e5d0 sp=0xc023a0e460 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88910, 0xc043a04910, 0xc023a0e840, 0xc00578cc00)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0e740 sp=0xc023a0e5d0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e888c0, 0xc043a04910, 0xc023a0e9b0, 0xc00578b540)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0e8b0 sp=0xc023a0e740 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88870, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0ea20 sp=0xc023a0e8b0 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88780, 0xc043a04910, 0x0, 0x0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0eb90 sp=0xc023a0ea20 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88af0, 0xc043a04910, 0xc023a0ee00, 0xc00578b4c0)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0ed00 sp=0xc023a0eb90 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88aa0, 0xc043a04910, 0xc023a0ef70, 0xc00578cb70)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0ee70 sp=0xc023a0ed00 pc=0xba1c0f
github.com/moby/buildkit/cache/remotecache/v1.marshalItem(0xc000e88a50, 0xc043a04910, 0xc023a0f0e0, 0xc00578cb40)
    /src/cache/remotecache/v1/utils.go:282 +0x16f fp=0xc023a0efe0 sp=0xc023a0ee70 pc=0xba1c0f
...additional frames elided...
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /src/vendor/google.golang.org/grpc/server.go:720 +0xa1

goroutine 1 [select, 4 minutes]:
main.main.func3(0xc0000ccb00, 0x0, 0x0)
    /src/cmd/buildkitd/main.go:258 +0x913
github.com/urfave/cli.HandleAction(0x1028e80, 0x1245768, 0xc0000ccb00, 0xc0000ccb00, 0x0)
    /src/vendor/github.com/urfave/cli/app.go:523 +0xbe
github.com/urfave/cli.(*App).Run(0xc0001c0000, 0xc0000320c0, 0xc, 0xc, 0x0, 0x0)
    /src/vendor/github.com/urfave/cli/app.go:285 +0x5ec
main.main()
    /src/cmd/buildkitd/main.go:285 +0xd4b

goroutine 18 [syscall, 4 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 13 [chan receive, 4 minutes]:
github.com/moby/buildkit/util/appcontext.Context.func1.1(0xc0001220c0, 0xc000104da0, 0xc00012b068)
    /src/util/appcontext/appcontext.go:30 +0x38
created by github.com/moby/buildkit/util/appcontext.Context.func1
    /src/util/appcontext/appcontext.go:28 +0xfe

goroutine 14 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc00007d790, 0xc00000001b)
    /usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc00007d780)
    /usr/local/go/src/sync/cond.go:56 +0x9d
github.com/moby/buildkit/util/cond.(*StatefulCond).Wait(0xc0000f10b0)
    /src/util/cond/cond.go:28 +0xaf
github.com/moby/buildkit/solver.(*scheduler).loop(0xc000393b20)
    /src/solver/scheduler.go:101 +0x18a
created by github.com/moby/buildkit/solver.newScheduler
    /src/solver/scheduler.go:35 +0x1ab

goroutine 29 [chan receive, 3 minutes]:
github.com/moby/buildkit/util/progress.pipe.func1(0x139afe0, 0xc000340380, 0xc0000f08d0)
    /src/util/progress/progress.go:167 +0x4c
created by github.com/moby/buildkit/util/progress.pipe
    /src/util/progress/progress.go:166 +0x12f

goroutine 51 [chan receive]:
github.com/moby/buildkit/util/pull.newResolverCache.func1(0xc0001e2090, 0xc0001320f0)
    /src/util/pull/resolver.go:174 +0x49
created by github.com/moby/buildkit/util/pull.newResolverCache
    /src/util/pull/resolver.go:172 +0x95

goroutine 53 [chan receive, 4 minutes]:
github.com/moby/buildkit/solver.(*scheduler).loop.func2(0xc000393b20)
    /src/solver/scheduler.go:76 +0x3c
created by github.com/moby/buildkit/solver.(*scheduler).loop
    /src/solver/scheduler.go:75 +0x77

goroutine 15 [IO wait]:
internal/poll.runtime_pollWait(0x7fef78561200, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00012d318, 0x72, 0x0, 0x0, 0x11f6d62)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00012d300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8
net.(*netFD).accept(0xc00012d300, 0xc001120518, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*UnixListener).accept(0xc0000f1830, 0xc0001b7d30, 0xc0001b7d38, 0x18)
    /usr/local/go/src/net/unixsock_posix.go:162 +0x32
net.(*UnixListener).Accept(0xc0000f1830, 0x1245318, 0xc0004d8c00, 0x13ab020, 0xc001120518)
    /usr/local/go/src/net/unixsock.go:260 +0x47
google.golang.org/grpc.(*Server).Serve(0xc0004d8c00, 0x1395ae0, 0xc0000f1830, 0x0, 0x0)
    /src/vendor/google.golang.org/grpc/server.go:597 +0x22e
main.serveGRPC.func1.1(0x0, 0x0)
    /src/cmd/buildkitd/main.go:322 +0x142
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0000f17a0, 0xc000289600)
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66

goroutine 16 [IO wait, 4 minutes]:
internal/poll.runtime_pollWait(0x7fef785612d0, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00012d498, 0x72, 0x0, 0x0, 0x11f6d62)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00012d480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8
net.(*netFD).accept(0xc00012d480, 0xc0004d6000, 0x7fef7d80bd98, 0x0)
    /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0002895c0, 0x40df48, 0x380, 0x11ee200)
    /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0002895c0, 0xc0001b3ce8, 0x45d4e0, 0xc0001b3d30, 0xc0001b3d38)
    /usr/local/go/src/net/tcpsock.go:261 +0x47
crypto/tls.(*listener).Accept(0xc0002895e0, 0x1245318, 0xc0004d8c00, 0x13aa7e0, 0xc000458000)
    /usr/local/go/src/crypto/tls/tls.go:59 +0x37
google.golang.org/grpc.(*Server).Serve(0xc0004d8c00, 0x138fe60, 0xc0002895e0, 0x0, 0x0)
    /src/vendor/google.golang.org/grpc/server.go:597 +0x22e
main.serveGRPC.func1.1(0x0, 0x0)
    /src/cmd/buildkitd/main.go:322 +0x142
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0000f17a0, 0xc000289620)
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66

goroutine 66 [semacquire, 4 minutes]:
sync.runtime_Semacquire(0xc0000f17b0)
    /usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc0000f17a8)
    /usr/local/go/src/sync/waitgroup.go:130 +0x64
golang.org/x/sync/errgroup.(*Group).Wait(0xc0000f17a0, 0x0, 0x0)
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x31
main.serveGRPC.func2(0xc00007fb00, 0xc0000f17a0)
    /src/cmd/buildkitd/main.go:327 +0x2b
created by main.serveGRPC
    /src/cmd/buildkitd/main.go:326 +0x39e

goroutine 86 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc0003403d0, 0xc0000001c8)
    /usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc0003403c0)
    /usr/local/go/src/sync/cond.go:56 +0x9d
github.com/moby/buildkit/util/progress.(*progressReader).Read(0xc0000f08d0, 0x139afe0, 0xc00011c3c0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /src/util/progress/progress.go:127 +0x12e
github.com/moby/buildkit/solver.(*Job).Status(0xc0003921c0, 0x139afe0, 0xc00011c3c0, 0xc0000ff620, 0x0, 0x0)
    /src/solver/progress.go:25 +0xd29
github.com/moby/buildkit/solver/llbsolver.(*Solver).Status(0xc000393ab0, 0x139afe0, 0xc00011c3c0, 0xc00012eb80, 0x19, 0xc0000ff620, 0xc000017fa8, 0x8f894b)
    /src/solver/llbsolver/solver.go:308 +0xbe
github.com/moby/buildkit/control.(*Controller).Status.func1(0xc000017f68, 0xc0000ea300)
    /src/control/control.go:300 +0x5e
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0002fb5c0, 0xc0002fb5f0)
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66

goroutine 98 [chan receive, 3 minutes]:
github.com/moby/buildkit/control.(*Controller).Session.func1(0xc0001d2000, 0xc000386500)
    /src/control/control.go:357 +0x34
created by github.com/moby/buildkit/control.(*Controller).Session
    /src/control/control.go:356 +0x16e

goroutine 85 [semacquire, 3 minutes]:
sync.runtime_Semacquire(0xc0002fb5d0)
    /usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc0002fb5c8)
    /usr/local/go/src/sync/waitgroup.go:130 +0x64
golang.org/x/sync/errgroup.(*Group).Wait(0xc0002fb5c0, 0xc000288620, 0x139afe0)
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x31
github.com/moby/buildkit/control.(*Controller).Status(0xc00036c480, 0xc0002fb320, 0x13a8500, 0xc0005379f0, 0xc00036c480, 0x20)
    /src/control/control.go:347 +0x1a3
github.com/moby/buildkit/api/services/control._Control_Status_Handler(0x1133300, 0xc00036c480, 0x13a3040, 0xc000288600, 0x119f980, 0x1b77100)
    /src/api/services/control/control.pb.go:1499 +0x109
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingStreamServerInterceptor.func1(0x1133300, 0xc00036c480, 0x13a3520, 0xc000032180, 0xc0002885a0, 0x1244a58, 0x0, 0x0)
    /src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:114 +0x378
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0004d8c00, 0x13ad780, 0xc0004d9e00, 0xc0000ea300, 0xc0000f1140, 0x1aeef40, 0x0, 0x0, 0x0)
    /src/vendor/google.golang.org/grpc/server.go:1244 +0x47a
google.golang.org/grpc.(*Server).handleStream(0xc0004d8c00, 0x13ad780, 0xc0004d9e00, 0xc0000ea300, 0x0)
    /src/vendor/google.golang.org/grpc/server.go:1317 +0xd30
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000296180, 0xc0004d8c00, 0x13ad780, 0xc0004d9e00, 0xc0000ea300)
    /src/vendor/google.golang.org/grpc/server.go:722 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /src/vendor/google.golang.org/grpc/server.go:720 +0xa1

goroutine 87 [chan receive]:
github.com/moby/buildkit/control.(*Controller).Status.func2(0xc000018f68, 0xc0002969e0)
    /src/control/control.go:305 +0x782
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0002fb5c0, 0xc000288620)
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
    /src/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66

goroutine 84 [chan receive, 3 minutes]:
github.com/moby/buildkit/session.(*Manager).handleConn(0xc00000f740, 0x139afe0, 0xc00007c740, 0x13aacc0, 0xc00010c280, 0xc0002facf0, 0x0, 0x0)
    /src/session/manager.go:144 +0x4c2
github.com/moby/buildkit/session.(*Manager).HandleConn(0xc00000f740, 0x139afe0, 0xc00007c4c0, 0x13aacc0, 0xc00010c280, 0xc0002facf0, 0x0, 0x30)
    /src/session/manager.go:97 +0x74
github.com/moby/buildkit/control.(*Controller).Session(0xc00036c480, 0x13aa900, 0xc0003864f0, 0x0, 0x0)
    /src/control/control.go:361 +0x1be
github.com/moby/buildkit/api/services/control._Control_Session_Handler(0x1133300, 0xc00036c480, 0x13a3040, 0xc0003a8da0, 0x119f980, 0x1b77100)
    /src/api/services/control/control.pb.go:1516 +0xad
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingStreamServerInterceptor.func1(0x1133300, 0xc00036c480, 0x13a3520, 0xc0001ca000, 0xc0003a8d40, 0x1244a48, 0x0, 0x0)
    /src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:114 +0x378
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0004d8c00, 0x13ad780, 0xc0004d9e00, 0xc0000ea200, 0xc0000f1140, 0x1aeef60, 0x0, 0x0, 0x0)
    /src/vendor/google.golang.org/grpc/server.go:1244 +0x47a
google.golang.org/grpc.(*Server).handleStream(0xc0004d8c00, 0x13ad780, 0xc0004d9e00, 0xc0000ea200, 0x0)
    /src/vendor/google.golang.org/grpc/server.go:1317 +0xd30
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000296180, 0xc0004d8c00, 0x13ad780, 0xc0004d9e00, 0xc0000ea200)
    /src/vendor/google.golang.org/grpc/server.go:722 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
    /src/vendor/google.golang.org/grpc/server.go:720 +0xa1

goroutine 100 [chan receive, 3 minutes]:
google.golang.org/grpc.(*addrConn).resetTransport(0xc0000d0000)
    /src/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ea
created by google.golang.org/grpc.(*addrConn).connect
    /src/vendor/google.golang.org/grpc/clientconn.go:815 +0x128

goroutine 81 [select, 3 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0004d9e00)
    /src/vendor/google.golang.org/grpc/internal/transport/http2_server.go:968 +0x26b
created by google.golang.org/grpc/internal/transport.newHTTP2Server
    /src/vendor/google.golang.org/grpc/internal/transport/http2_server.go:302 +0x1082

goroutine 80 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000132b90, 0x1, 0x0, 0x0, 0x0, 0x0)
    /src/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00007e6c0, 0x0, 0x0)
    /src/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0004d9e00)
    /src/vendor/google.golang.org/grpc/internal/transport/http2_server.go:296 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
    /src/vendor/google.golang.org/grpc/internal/transport/http2_server.go:293 +0x105d

goroutine 82 [IO wait]:
internal/poll.runtime_pollWait(0x7fef78561130, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc00012c218, 0x72, 0x9200, 0x921e, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00012c200, 0xc000ab6000, 0x921e, 0x921e, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc00012c200, 0xc000ab6000, 0x921e, 0x921e, 0x203001, 0x422985, 0xc005002080)
    /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000106020, 0xc000ab6000, 0x921e, 0x921e, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x68
crypto/tls.(*atLeastReader).Read(0xc005002080, 0xc000ab6000, 0x921e, 0x921e, 0xeb, 0x1375b00, 0xc0000619b0)
    /usr/local/go/src/crypto/tls/conn.go:780 +0x60
bytes.(*Buffer).ReadFrom(0xc000458258, 0x13759e0, 0xc005002080, 0x40b6b5, 0x108c0a0, 0x11c1c40)
    /usr/local/go/src/bytes/buffer.go:204 +0xb4
crypto/tls.(*Conn).readFromUntil(0xc000458000, 0x1377260, 0xc000106020, 0x5, 0xc000106020, 0x7fef70268200)
    /usr/local/go/src/crypto/tls/conn.go:802 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc000458000, 0x0, 0x0, 0xc005004000)
    /usr/local/go/src/crypto/tls/conn.go:609 +0x124
crypto/tls.(*Conn).readRecord(...)
    /usr/local/go/src/crypto/tls/conn.go:577
crypto/tls.(*Conn).Read(0xc000458000, 0xc000590000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1255 +0x161
bufio.(*Reader).Read(0xc0000fe8a0, 0xc0001be118, 0x9, 0x9, 0x2, 0x1357b4c, 0x16db6db6db6db6d)
    /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x1375820, 0xc0000fe8a0, 0xc0001be118, 0x9, 0x9, 0x9, 0xc000b68000, 0x11, 0x11)
    /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0001be118, 0x9, 0x9, 0x1375820, 0xc0000fe8a0, 0x0, 0x0, 0x11002fa1e0, 0x1100000001)
    /src/vendor/golang.org/x/net/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001be0e0, 0xc0002fa1e0, 0x1b54200, 0x0, 0x0)
    /src/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0004d9e00, 0xc0002fa120, 0x1245350)
    /src/vendor/google.golang.org/grpc/internal/transport/http2_server.go:458 +0xa1
google.golang.org/grpc.(*Server).serveStreams(0xc0004d8c00, 0x13ad780, 0xc0004d9e00)
    /src/vendor/google.golang.org/grpc/server.go:718 +0xf3
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc0004d8c00, 0x13ad780, 0xc0004d9e00)
    /src/vendor/google.golang.org/grpc/server.go:679 +0x3f
created by google.golang.org/grpc.(*Server).handleRawConn
    /src/vendor/google.golang.org/grpc/server.go:678 +0x562

goroutine 28 [select, 3 minutes]:
main.unaryInterceptor.func1.1(0x139afe0, 0xc000340080, 0x139afe0, 0xc00007cfc0, 0xc0000f4080)
    /src/cmd/buildkitd/main.go:524 +0xd8
created by main.unaryInterceptor.func1
    /src/cmd/buildkitd/main.go:523 +0x121

goroutine 99 [select, 3 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00007c6c0)
    /src/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc2
created by google.golang.org/grpc.newCCBalancerWrapper
    /src/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16d

goroutine 101 [select]:
github.com/moby/buildkit/session.monitorHealth(0x139afe0, 0xc00007c740, 0xc000057880, 0xc000386630)
    /src/session/grpc.go:69 +0x1cb
created by github.com/moby/buildkit/session.grpcClientConn
    /src/session/grpc.go:55 +0x270

goroutine 103 [chan receive, 3 minutes]:
github.com/moby/buildkit/util/progress.pipe.func1(0x139afe0, 0xc00007c800, 0xc0003e8b70)
    /src/util/progress/progress.go:167 +0x4c
created by github.com/moby/buildkit/util/progress.pipe
    /src/util/progress/progress.go:166 +0x12f

goroutine 89 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0000b22d0, 0xc0003a8d30, 0x5, 0x5, 0x7fef7b5db881, 0x10, 0x1246140)
    /src/vendor/google.golang.org/grpc/internal/transport/transport.go:174 +0xc7
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0000b22d0, 0xc0003a8d30, 0x5, 0x5, 0xc000c07b38, 0xc00012eb00, 0xc000c07818)
    /src/vendor/google.golang.org/grpc/internal/transport/transport.go:168 +0x21a
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0002faf00, 0xc0003a8d30, 0x5, 0x5, 0xc0006080c0, 0xc000c07840, 0x8c055c)
    /src/vendor/google.golang.org/grpc/internal/transport/transport.go:479 +0x55
io.ReadAtLeast(0x1377020, 0xc0002faf00, 0xc0003a8d30, 0x5, 0x5, 0x5, 0xc0006080b2, 0xa, 0x0)
    /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0000ea200, 0xc0003a8d30, 0x5, 0x5, 0xc0005c2000, 0x1088ca0, 0x294a5294a5)
    /src/vendor/google.golang.org/grpc/internal/transport/transport.go:463 +0xcd
google.golang.org/grpc.(*parser).recvMsg(0xc0003a8d20, 0x400000, 0xe, 0xc000c079f0, 0x40d8c4, 0xc004c8c2a0, 0x30, 0x20)
    /src/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63
google.golang.org/grpc.recvAndDecompress(0xc0003a8d20, 0xc0000ea200, 0x0, 0x0, 0x400000, 0x0, 0x0, 0x0, 0xc0006080e0, 0x1357b4c, ...)
    /src/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d
google.golang.org/grpc.recv(0xc0003a8d20, 0x7fef78d99240, 0x1b77100, 0xc0000ea200, 0x0, 0x0, 0x11a0600, 0xc0009e6f40, 0x400000, 0x0, ...)
    /src/vendor/google.golang.org/grpc/rpc_util.go:707 +0x9b
google.golang.org/grpc.(*serverStream).RecvMsg(0xc0001ca000, 0x11a0600, 0xc0009e6f40, 0x0, 0x0)
    /src/vendor/google.golang.org/grpc/stream.go:1474 +0x170
github.com/moby/buildkit/session/grpchijack.(*conn).Read(0xc00010c280, 0xc000234000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /src/session/grpchijack/dial.go:69 +0x21e
bufio.(*Reader).Read(0xc0000ff9e0, 0xc0001fa038, 0x9, 0x9, 0x20, 0xc000c07d00, 0x8ac82c)
    /usr/local/go/src/bufio/bufio.go:226 +0x26a
io.ReadAtLeast(0x1375820, 0xc0000ff9e0, 0xc0001fa038, 0x9, 0x9, 0x9, 0x5a525d85efee8601, 0x0, 0x0)
    /usr/local/go/src/io/io.go:310 +0x87
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0001fa038, 0x9, 0x9, 0x1375820, 0xc0000ff9e0, 0x0, 0x0, 0x0, 0x0)
    /src/vendor/golang.org/x/net/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001fa000, 0xc004c8c270, 0xc004c8c270, 0x0, 0x0)
    /src/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0001fc000)
    /src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x183
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd23

goroutine 90 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000b24b0, 0x1, 0x0, 0x0, 0x0, 0x0)
    /src/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x122
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00029a000, 0x0, 0x0)
    /src/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e3
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0001fc000)
    /src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
    /src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedb

goroutine 30 [select, 3 minutes]:
github.com/moby/buildkit/util/progress.(*MultiReader).Reader.func1(0x139b0a0, 0xc0000f0990, 0xc0003e8c60, 0xc00000e360)
    /src/util/progress/multireader.go:37 +0xc1
created by github.com/moby/buildkit/util/progress.(*MultiReader).Reader
    /src/util/progress/multireader.go:36 +0x1c2

goroutine 31 [sync.Cond.Wait]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc00007c850, 0xc000000239)
    /usr/local/go/src/runtime/sema.go:510 +0xf8
sync.(*Cond).Wait(0xc00007c840)
    /usr/local/go/src/sync/cond.go:56 +0x9d
github.com/moby/buildkit/util/progress.(*progressReader).Read(0xc0003e8b70, 0x139b020, 0xc00003c0a8, 0x0, 0x0, 0x0, 0x0, 0x0)
    /src/util/progress/progress.go:127 +0x12e
github.com/moby/buildkit/util/progress.(*MultiReader).handle(0xc0003e8c60, 0xc0000f0990, 0xc0003e8c60)
    /src/util/progress/multireader.go:56 +0x11d
created by github.com/moby/buildkit/util/progress.(*MultiReader).Reader
    /src/util/progress/multireader.go:47 +0x220

goroutine 11041 [select]:
github.com/moby/buildkit/util/progress.(*progressReader).Read.func1(0xc0014caf00, 0x139afe0, 0xc00011c3c0, 0xc0000f08d0)
    /src/util/progress/progress.go:101 +0xb7
created by github.com/moby/buildkit/util/progress.(*progressReader).Read
    /src/util/progress/progress.go:100 +0xf5

goroutine 11040 [select]:
github.com/moby/buildkit/util/progress.(*progressReader).Read.func1(0xc0014cade0, 0x139b020, 0xc00003c0a8, 0xc0003e8b70)
    /src/util/progress/progress.go:101 +0xb7
created by github.com/moby/buildkit/util/progress.(*progressReader).Read
    /src/util/progress/progress.go:100 +0xf5
[rootlesskit:child ] error: command [buildkitd --addr unix:///run/user/1000/buildkit/buildkitd.sock --addr tcp://0.0.0.0:1234 --tlscacert /certs/ca.pem --tlscert /certs/cert.pem --tlskey /certs/key.pem --oci-worker-no-process-sandbox] exited: exit status 2
[rootlesskit:parent] error: child exited: exit status 2
tonistiigi commented 4 years ago

@myhau Did you see this as a regression on v0.7.2 or not?

tonistiigi commented 4 years ago

@myhau I can't rebuild your example Dockerfile as I don't have access to the build context it is using. If this is a public project please post it. If not, try to make a repro with a public repo.

myhau commented 4 years ago

@tonistiigi this also happened on 0.7.1. It's a private project and I won't be able to share it. I will try to create some reproducer, though.

myhau commented 4 years ago

I won't be able to share a reproducer right now, but I started debugging this issue. I've added some print statements in cache/remotecache/v1/utils.go

func marshalItem(it *item, state *marshalState) error {
    if _, ok := state.recordsByItem[it]; ok {
        return nil
    }

    println("Item")
    println(it.dgst)
    println(len(it.links))

and thee seems to be a cycle:

... { SOME LAYERS } ...
Item
sha256:9cb10fb741d8a44ea802a711c50e7dd9544bd5a314d7d6a7ad1c74cb189b5795
2
Item
sha256:f31b0de45deb848d770f80588a1e48871b4192b51635eee9d96b8243949dc4b9
2
Item
sha256:29077b3fcacf630c03e045f74ceeece9ea7fed8951f9b8aea234954c171fa16f
2
Item
sha256:a81904108def4c8528d4f62a8b4697f865c5e946306c37d7c6c581a81d99c6a2
1
Item
sha256:4e32986c7015560b427bc34e8db5a5deeee1b94376274877f0b49ac1358d4553
2
Item
sha256:a0322fc9963e65b16a78bdb99030922d94b1984cdb5e402d0dc79a6760b4dbc0
2
Item
sha256:f7476d78c0ce02326f00e99553b3d19e81e96dae909830c65f6daca3c5e0953d
1
Item
sha256:1b56f72ea87be6c48fc070e678b01dcd7f5b8952b393c9b18037f6da7342b46d
2
Item
sha256:6a2f126cff983e4f7fc129a5d1a5c1ed91861bb522fd5c8b93aa12a104f7f6ad
2
Item
sha256:b0e3a2ffc9b52bfbd2be6188ca765a155af08a947158f8b844d29bd19a44a0c2
1
Item
sha256:9cb10fb741d8a44ea802a711c50e7dd9544bd5a314d7d6a7ad1c74cb189b5795
2
... { CYCLE REPEATED MANY TIMES } ...

Maybe this is related to https://github.com/moby/buildkit/pull/1382 / https://github.com/moby/buildkit/pull/1514 ?

tonistiigi commented 4 years ago

1382 was already picked to v0.7.2

tonistiigi commented 3 years ago

2410 looks related