krasi-georgiev
commented
5 years ago not sure if it will help but when I cancel - CTRK + C it returns with:
failed to receive status: rpc error: code = Canceled desc = context canceled
Closed krasi-georgiev closed 5 years ago
krasi-georgiev
commented
5 years ago not sure if it will help but when I cancel - CTRK + C it returns with:
failed to receive status: rpc error: code = Canceled desc = context canceled
tonistiigi
commented
5 years ago it's hard to see from the logs when it actually get stuck. I also don't see any logs for linux/arm/v7 at all in that output. Are you saying the build works fine with just --platform linux/arm/v7 but doesn't start for arm if you specify amd64 as well?
krasi-georgiev
commented
5 years ago I just tried it with only:
docker buildx build --platform linux/arm/v7 -t arribada/gateway --push --progress=plain .and it doesn't start at all. No logs output
can I enable some debug logs?
krasi-georgiev
commented
5 years ago I just tried to delete and recreate this context and the buildx ls command doesn't even see it as running
$ docker buildx rm
$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
default * docker
default default running linux/amd64
$ docker buildx create --append --name multi ssh://root@loraserver.local
WARN[0000] failed to find "multi" for append, creating a new instance instead
multi
$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
multi * docker-container
multi0 ssh://root@loraserver.local inactive
default docker
default default running linux/amd64
$ docker buildx inspect multi --bootstrap
[+] Building 4.5s (1/1) FINISHED
WARN[0005] commandConn.CloseWrite: commandconn: failed to wait: signal: killed 4.5s
=> ERROR [internal] booting buildkit 4.5s
=> => pulling image moby/buildkit:master 4.5s
------
> [internal] booting buildkit:
------
Name: multi
Driver: docker-container
Nodes:
Name: multi0
Endpoint: ssh://root@loraserver.local
Status: inactive
Platforms:
$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
multi * docker-container
multi0 ssh://root@loraserver.local inactive
default docker
default default running linux/amd64
Are you sure that endpoint is repsonsive? DOCKER_HOST=ssh://root@loraserver.local docker ps and look for a container in that list after --bootstrap reports it running.
krasi-georgiev
commented
5 years ago Yes it is responsive
I think I found some usefull logs by ssh-ing in the rpi and looking at the logs of the only container there:
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
de76d9247c21 moby/buildkit:master "buildkitd" 24 seconds ago Exited (2) 11 seconds ago buildx_buildkit_multi0
root@loraserver:~# docker logs de76d9247c21
time="2019-08-05T23:55:41Z" level=warning msg="using host network as the default"
time="2019-08-05T23:55:41Z" level=info msg="found worker \"vje74b849h7h4t08bfef44tf3\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:de76d9247c21 org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/arm/v7 linux/arm/v6]"
time="2019-08-05T23:55:41Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2019-08-05T23:55:41Z" level=info msg="found 1 workers, default=\"vje74b849h7h4t08bfef44tf3\""
time="2019-08-05T23:55:41Z" level=warning msg="currently, only the default worker can be used."
time="2019-08-05T23:55:41Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x126f0]
goroutine 49 [running]:
github.com/moby/buildkit/control.(*Controller).Solve(0x31daf00, 0xc08e70, 0x3372f60, 0x336e510, 0x0, 0x0, 0x0)
/src/control/control.go:217 +0x54
github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1(0xc08e70, 0x3372f60, 0xa73ac0, 0x336e510, 0xa60f78, 0x1318378, 0xc08e70, 0x3372f60)
/src/api/services/control/control.pb.go:1364 +0x68
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0xc08e10, 0x3372ee0, 0xa73ac0, 0x336e510, 0x327b4b0, 0x327b4c0, 0x0, 0x0, 0xbfbb90, 0x307c8b0)
/src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57 +0x250
main.unaryInterceptor.func1(0xc08e10, 0x3372ee0, 0xa73ac0, 0x336e510, 0x327b4b0, 0x327b4c0, 0x0, 0x0, 0x0, 0x0)
/src/cmd/buildkitd/main.go:526 +0xfc
github.com/moby/buildkit/api/services/control._Control_Solve_Handler(0xa3d100, 0x31daf00, 0xc08e70, 0x3372e80, 0x33a82d0, 0x327ad90, 0xc08e70, 0x3372e80, 0x319a480, 0xb7)
/src/api/services/control/control.pb.go:1366 +0x118
google.golang.org/grpc.(*Server).processUnaryRPC(0x3100c00, 0xc10d48, 0x31281e0, 0x336e3f0, 0x33724a0, 0x12ff784, 0x0, 0x0, 0x0)
/src/vendor/google.golang.org/grpc/server.go:972 +0x484
google.golang.org/grpc.(*Server).handleStream(0x3100c00, 0xc10d48, 0x31281e0, 0x336e3f0, 0x0)
/src/vendor/google.golang.org/grpc/server.go:1252 +0xb18
google.golang.org/grpc.(*Server).serveStreams.func1.1(0x3028020, 0x3100c00, 0xc10d48, 0x31281e0, 0x336e3f0)
/src/vendor/google.golang.org/grpc/server.go:691 +0x7c
created by google.golang.org/grpc.(*Server).serveStreams.func1
/src/vendor/google.golang.org/grpc/server.go:689 +0x7cHmm, this looks like https://github.com/moby/buildkit/pull/1082
~Can you post DOCKER_HOST=ssh://root@loraserver.local docker inspect moby/buildkit:master | egrep "Id|Created"~
Same issue but a different struct. I'll submit a patch.
krasi-georgiev
commented
5 years ago thanks, let me know when you push the new image to moby/buildkit and I will test again.
krasi-georgiev
commented
5 years ago ok I see the change has been merged. When would the new master image be pushed?
tonistiigi
commented
5 years ago @krasi-georgiev The master build is automatic and the new image should have the fix
FernandoMiguel
commented
5 years ago @tonistiigi the builders have been broken since the 27th of last month sadly
krasi-georgiev
commented
5 years ago unfortunately I just tried it and still panics:
I deleted the image to make sure it pulls the last one, but not sure if it really is the last one as it shows that it was created 35h ago
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
moby/buildkit master 5a72ed64a203 35 hours ago 68.1MB# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9f8d9e697672 moby/buildkit:master "buildkitd" 13 seconds ago Created buildx_buildkit_multi0
# docker logs -f 9f8d9e697672
time="2019-08-07T09:58:07Z" level=warning msg="using host network as the default"
time="2019-08-07T09:58:09Z" level=info msg="found worker \"emlwaf149boyp27ixuf1oxca0\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:9f8d9e697672 org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/arm/v7 linux/arm/v6]"
time="2019-08-07T09:58:09Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2019-08-07T09:58:09Z" level=info msg="found 1 workers, default=\"emlwaf149boyp27ixuf1oxca0\""
time="2019-08-07T09:58:09Z" level=warning msg="currently, only the default worker can be used."
time="2019-08-07T09:58:09Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x126f0]
goroutine 20 [running]:
github.com/moby/buildkit/control.(*Controller).Solve(0x35eaf00, 0xc08e70, 0x34d9060, 0x37ca360, 0x0, 0x0, 0x0)
/src/control/control.go:217 +0x54
github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1(0xc08e70, 0x34d9060, 0xa73ac0, 0x37ca360, 0xa60f78, 0x1318378, 0xc08e70, 0x34d9060)
/src/api/services/control/control.pb.go:1364 +0x68
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0xc08e10, 0x34d8fe0, 0xa73ac0, 0x37ca360, 0x34da370, 0x34da380, 0x0, 0x0, 0xbfbb90, 0x340e868)
/src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57 +0x250
main.unaryInterceptor.func1(0xc08e10, 0x34d8fe0, 0xa73ac0, 0x37ca360, 0x34da370, 0x34da380, 0x0, 0x0, 0x0, 0x0)
/src/cmd/buildkitd/main.go:526 +0xfc
github.com/moby/buildkit/api/services/control._Control_Solve_Handler(0xa3d100, 0x35eaf00, 0xc08e70, 0x34d8f80, 0x34d6870, 0x3684d90, 0xc08e70, 0x34d8f80, 0x37fe180, 0xb7)
/src/api/services/control/control.pb.go:1366 +0x118
google.golang.org/grpc.(*Server).processUnaryRPC(0x350ac00, 0xc10d48, 0x37fc000, 0x37ca1b0, 0x3774320, 0x12ff784, 0x0, 0x0, 0x0)
/src/vendor/google.golang.org/grpc/server.go:972 +0x484
google.golang.org/grpc.(*Server).handleStream(0x350ac00, 0xc10d48, 0x37fc000, 0x37ca1b0, 0x0)
/src/vendor/google.golang.org/grpc/server.go:1252 +0xb18
google.golang.org/grpc.(*Server).serveStreams.func1.1(0x37b0060, 0x350ac00, 0xc10d48, 0x37fc000, 0x37ca1b0)
/src/vendor/google.golang.org/grpc/server.go:691 +0x7c
created by google.golang.org/grpc.(*Server).serveStreams.func1
/src/vendor/google.golang.org/grpc/server.go:689 +0x7c@tonistiigi if you give me your pub key or let me know where to ping you on irc I will give you access to the RPI so It would be easier to find the culprit.
tonistiigi
commented
5 years ago @krasi-georgiev https://github.com/tonistiigi.keys
tonistiigi
commented
5 years ago @krasi-georgiev Seems lot of builds failed because of connection issues for github and this patch wasn't in the build you seem to be using(based on that timestamp). I restarted it and will let you know.
tonistiigi
commented
5 years ago @krasi-georgiev The build is complete now and everything seemed to be working in your server.
krasi-georgiev
commented
5 years ago Magic! thanks a lot!
DmitryMaksakov
commented
4 years ago I have the same problem: buildx is stucking at pushing layers. On ctrl-c I have failed to solve: rpc error: code = Canceled desc = context canceled. The problem occurs only if linux/arm/v7 in the list.
sredni
commented
4 years ago Just faced the same problem, how did you fixed it?
UPDATE Ok, I found that I have one quite big layer and it simply takes time to push (235.3s). Unfortunately push doesn’t report layer progress :(
I created a build context to use my local machine amd64 and a remote Rpi arm/v7
The build process seems to gets stuck at pushing the image.
It is a simple golang app