Getting "Error: [Sandbox] PipeToSandbox is closed" on self hosted grist

[JomerDev]

Hello,

I'm trying to run the gristlabs/grist:latest docker image. With pynbox I get the[Sandbox] PipeToSandbox is closed error. I looked at issue: https://github.com/gristlabs/grist-core/issues/293 however setting the sandbox flavor to gvisor and sandbox to "sandbox/gvisor/run.py" only results in a bad gateway error in the frontend the following error in the backend:

Sandbox stderr: run.py: sandbox/gvisor/run.py -E PYTHONPATH=/grist/sandbox/grist -E PIPE_MODE=minimal -m /grist/sandbox --checkpoint /tmp/engine__grist python3 -- /grist/sandbox/grist/main.py sandboxPid=23, flavor=gvisor, command=sandbox/gvisor/run.py, entryPoint=(default)

Leaving sandbox (the command) empty results in the same error.

I'm not sure how I can sandbox grists python formulas in docker

[paulfitz]

Hi @JomerDev, pynbox isn't included in that docker image, since it only supports Python 2.7.

There shouldn't be any need for you to set any paths for gvisor, can you try just this:

docker run -e GRIST_SANDBOX_FLAVOR=gvisor -p 8484:8484 -it gristlabs/grist:latest

[JomerDev]

Sure. The following is all that is printed out

$ docker run -e GRIST_SANDBOX_FLAVOR=gvisor -p 8484:8484 -it gristlabs/grist:latest
2022-12-19 14:49:13.780 - debug: 3-pipe Sandbox started sandboxPid=22, flavor=gvisor, command=undefined, entryPoint=(default)
2022-12-19 14:49:14.167 - info: Sandbox stderr: run.py: sandbox/gvisor/run.py -E PYTHONPATH=/grist/sandbox/grist -E PIPE_MODE=minimal -m /grist/sandbox --checkpoint /tmp/engine__grist python3 -- /grist/sandbox/grist/main.py sandboxPid=22, flavor=gvisor, command=undefined, entryPoint=(default)

[paulfitz]

Thanks, looks like the sandbox is not working for you. Those lines are Grist trying to build a "checkpoint", and then hanging. Regular behavior would look like:

2022-12-19 15:25:47.986 - debug: 3-pipe Sandbox started sandboxPid=24, flavor=gvisor, command=undefined, entryPoint=(default)
2022-12-19 15:25:48.008 - info: Sandbox stderr: run.py: sandbox/gvisor/run.py -E PYTHONPATH=/grist/sandbox/grist -E PIPE_MODE=minimal -m /grist/sandbox --checkpoint /tmp/engine__grist python3 -- /grist/sandbox/grist/main.py sandboxPid=24, flavor=gvisor, command=undefined, entryPoint=(default)
2022-12-19 15:25:48.854 - info: Sandbox stderr: Ready message: [INFO] [__main__] Ready sandboxPid=24, flavor=gvisor, command=undefined, entryPoint=(default)
2022-12-19 15:25:49.011 - warn: Sandbox unexpectedly exited with code 0 signal null sandboxPid=24, flavor=gvisor, command=undefined, entryPoint=(default)
Welcome to Grist.
In quiet mode, see http://localhost:8484 to use.
For full logs, re-run with DEBUG=1
2022-12-19 15:25:50.064 - info: == Grist version is 1.0.5 (commit unknown)

What kind of host is the container running on? Gvisor-based sandboxing has been reported to fail to initialize on older Intel processors that do not support the XSAVE feature (supported by Sandy Bridge and later).

[JomerDev]

It's an Odroid-M1

[paulfitz]

Hmm the docker image contains a version for ARM processors, I don't know if there are nuances on the range of processors supported - @georgegevoian may know more. If I were debugging this myself, I'd first check if gvisor (https://github.com/google/gvisor/) runs on your hardware.

[JomerDev]

I have installed and can start gvisors runsc just fine. But I have no project to test it with. Do the grist tests run the sandbox?

[paulfitz]

Can you check if sandbox/gvisor/run.py bash starts up a shell? If that works, can you check the output of echo "import sys; print(sys.version)" | sandbox/gvisor/run.py python3 ?

If runsc is working for you, then we should be able to get Grist working by building from source and setting the right env vars. It might take some work.

[JomerDev]

In the docker container or outside of it?

I assumed inside of the docker container. There both don't seem to work. They print out one line (run.py: ./run.py bash or run.py: ./run.py python3) but don't do anything after that. I can stop the with Ctrl + C though

[georgegevoian]

Hi @JomerDev.

I tried running the latest arm64 image with gvisor and didn't run into any hangs or other issues. I'm not sure why it's happening, but we can dig deeper into the logs and see if they reveal anything. The command below should enable debug logging, putting all log files in /tmp/. Can you see if any of them have warnings or errors?

docker run -p 8484:8484 -it --env GRIST_SANDBOX_FLAVOR=gvisor --env GVISOR_FLAGS="-unprivileged -ignore-cgroups -debug -strace -debug-log=/tmp/" gristlabs/grist:latest

[JomerDev]

I've attached the log files it generated here: runsc-logs.zip Github wouldn't let me upload them individually

Sadly none of them seem to include any errors as far as I can tell

Edit: After trying this a few times I saw that when I run this, one runsc-sandbox process takes up 100% of one core continuously until I stop the docker container again

[JomerDev]

@georgegevoian @paulfitz Debugging this a bit more, I found that runsc-sandbox is busy looping calling munmap. I've opened an issue there with more information regarding the issue. I don't think it has much to do with grist at all