openziti / zrok

Geo-scale, next-generation peer-to-peer sharing platform built on top of OpenZiti.
https://zrok.io
Apache License 2.0
2.47k stars 96 forks source link

zrok enable Error Path Crash #244

Closed michaelquigley closed 1 year ago

michaelquigley commented 1 year ago
[WARNING]: unable to open zrokdir metadata; ignoring

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xf44e62]

goroutine 1 [running]:
github.com/charmbracelet/bubbletea.(*Program).Send(...)
        /home/runner/go/pkg/mod/github.com/charmbracelet/bubbletea@v0.23.1/tea.go:501
main.(*enableCommand).run(0xc0000133e0, 0x0?, {0xc0004881b0, 0x1, 0x0?})
        /home/runner/work/zrok/zrok/cmd/zrok/enable.go:96 +0x642
github.com/spf13/cobra.(*Command).execute(0xc00041e500, {0xc000488190, 0x1, 0x1})
^[[36;1R        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876 +0x67b
github.com/spf13/cobra.(*Command).ExecuteC(0x2c85ac0)
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
main.main()
        /home/runner/work/zrok/zrok/cmd/zrok/main.go:86 +0x25
qrkourier commented 1 year ago

When this happens the enabled environment appears in the controller UI, but ~/.zrok is still empty. I made sure the same user is able to touch a file in filesystem that's mounted on ~/.zrok.

qrkourier commented 1 year ago

I built the branch associated with this issue and can reproduce this stack trace. I assume this is being triggered when zrok tries to write files like ~/.zrok/environment.json, and that there's something about the way the storage is being provided that preventing that.

[WARNING]: unable to open zrokdir metadata; ignoring

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xf4504d]

goroutine 1 [running]:
github.com/charmbracelet/bubbletea.(*Program).Send(...)
        /usr/share/go/pkg/mod/github.com/charmbracelet/bubbletea@v0.23.1/tea.go:501
main.(*enableCommand).run(0xc0000133b0, 0x0?, {0xc0004841e0, 0x1, 0x0?})
        /mnt/cmd/zrok/enable.go:107 +0x84d
github.com/spf13/cobra.(*Command).execute(0xc000414500, {0xc0004841c0, 0x1, 0x1})
        /usr/share/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876 +0x67b
github.com/spf13/cobra.(*Command).ExecuteC(0x2c8ea80)
        /usr/share/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /usr/share/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
main.main()
        /mnt/cmd/zrok/main.go:86 +0x25

Steps to reproduce on linux-amd64:

(set -eux;
git checkout docker-share  # build a container image with the right libc 
docker buildx build -t zrok-builder ./docker-images/cross-build --load
git checkout enable_error_path  # this is Michael's fix branch
rm -rf ./dist && docker run --rm --volume $PWD:/mnt zrok-builder amd64  # this runs the builder container to create the zrok artifact in ./dist/
sha256sum ./dist/amd64/linux/zrok  # note checksum
git checkout docker-share  # this branch has the container stuff to reproduce
pushd ./docker-images/zrok-public-share/
docker compose build zrok-enable  # stuff the dev build into the zrok-enable image
docker compose run --entrypoint /bin/bash zrok-enable -c "sha256sum /usr/local/bin/zrok"  # verify checksum
read -s ZROK_ENABLE_TOKEN; export ZROK_ENABLE_TOKEN  # paste token and press ENTER
docker compose down --remove-orphans --volumes && docker compose up --no-log-prefix zrok-enable  # run the modified container
docker compose run --entrypoint /bin/bash zrok-enable -c 'echo WHOAMI=$(id); touch /mnt/.zrok/touchtest; ls -lAR /mnt/'
;)
qrkourier commented 1 year ago

@michaelquigley with latest dev build from your commit I now see this message during enable. The enable process consistently gets stuck at "contacting the zrok service...", the same errors I reported in a DM are emitted in the controller UI's JS console, there's no longer a stack trace, and the controller UI doesn't show the enabled environment succeeded like it was doing with the prior build.

[   0.434]   ERROR main.(*enableCommand).run: the zrok service returned an error: Post "https://api.staging.zrok.io/api/v1/enable": net/http: invalid header field value for "X-Token"