Open ctalledo opened 4 years ago
ctalledo
commented
4 years ago Another hint: I noticed that if I launch multiple sys containers with vagrant, and then destroy them, the destruction goes well for all but the last remaining sys container. In the prior comment I speculated on the shiftfs mount being the culprit, and this latest observation adds weight to that since shiftfs would only be unmounted when destroying the last remaining container.
ctalledo
commented
4 years ago I also noticed another problem: when provisioning 16 sys containers with Vagrant (using the Docker provider with Sysbox), normally one of the containers fails to provision, with the following error:
An error occurred while executing multiple actions in parallel.
Any errors that occurred are shown below.
An error occurred while executing the action on the 'server14'
machine. Please handle this error then try again:
A Docker command executed by Vagrant didn't complete successfully!
The command run along with the output from the command is shown
below.
Command: ["docker", "run", "--name", "syscont_server14_1600385625", "-d", "-v", "/home/ctalledo/vm-vs-syscont-perf/provision-time/syscont:/vagrant", "--runtime=sysbox-runc", "--cpus=2", "-m=2g", "benchmark-syscont", {:notify=>[:stdout, :stderr]}]
Stderr: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
/usr/bin/docker: Error response from daemon: OCI runtime create failed: error in the container spec: invalid mount config: failed to request mounts from sysbox-mgr: failed to invoke ReqMounts via grpc: rpc error: code = DeadlineExceeded desc = context deadline exceeded: unknown.
Stdout: b151c7e0164214b46f4f058a6a33d50f1e95c5acd7e99616f979b0082850afe5 It seems that sysbox-runc is complaining that sysbox-mgr has taken too long to respond to a request issued via grpc. We need to investigate why sysbox-mgr is taking long enough to exceed the grpc deadline, and whether the grpc deadline is set correctly or should be increased.
Playing around with Vagrant, I was easily able to create a container using the Docker provider + Sysbox. Here is the Vagrant file:
Running
vagrant upworks without problem.However,
vagrant haltorvagrant destroy -ffail with a cryptic error:I am not quite sure what's going on, but in Linux this type of error generally occurs when a command is executed from within a directory that no longer exists. That does not appear to be the case here, at least on the surface.
Speculating here: I noticed that when Vagrant runs the container, it mounts the Vagrant file in the host into the container's
/vagrantdirectory. This causes Sysbox to mountshiftfson the host directory where the Vagrant file is located. This in turn makes that directory non-exec. I wonder if this is playing a role somehow.Also: after the error occurs, the container is stopped but not removed. It must be removed explicitly with "docker rm".