Open dhrp opened 6 years ago
@dhrp nice catch ! The culprit you're pointing here: https://github.com/containers/virtcontainers/blob/54b8c0cc68933561bc947c8b76aa491bb925f3ae/pkg/hyperstart/hyperstart.go#L345 is clearly a legacy constraint. We don't rely on the same agent anymore and there is no reason to keep this check IMO. Could you first test that it works for you if you remove this part, and if that's the case, could you submit a proper PR to fix this issue ?
Perhaps you can give me some pointers to how to include / replace this, and rebuild as it is not obvious to me where the source comes from / how it gets into the pkg directory.
I'm on #clearcontainers irc as thatcher
Hi @dhrp - please don't feel you have to raise a PR, but if you'd like to contribute, that would be awesome.
What you'll need to do is:
go get -d github.com/clearcontainers/runtime
the runtime as documented there.cd $GOPATH/src/github.com/clearcontainers/runtime
./vendor/github.com/containers/virtcontainers/pkg/hyperstart/hyperstart.go
to remove the check for 10240
.cc-runtime
binary will be installed as /usr/local/bin/cc-runtime
. That's probably what you want so you don't end up stamping over the packaged runtime (at /usr/bin/cc-runtime
). However, if the runtime is installed to /usr/local/bin/cc-runtime
, you'll need to update the docker config:
/etc/systemd/system/docker.service.d/clear-containers.conf
to specify cc-runtime=/usr/local/bin/cc-runtime
docker run -e ...
and see what happens.If that all works, you can then raise a PR on virtcontainers (not the runtime) to remove the 10240
check at: https://github.com/containers/virtcontainers/pulls
Once that fix lands in virtcontainers, we can then raise a PR on the runtime to "re-vendor" (update) the version of virtcontainers used by the runtime. At that point the bug will have been fully fixed.
I've raised an issue on our tests
repository so we don't forget to create a test to avoid this problem re-occurring: https://github.com/clearcontainers/tests/issues/817.
I'm also aware that the dev doc referenced above is missing the docker config instructions wrt the runtime I've outlined above, so I'll raise a PR to get that added...
Dev guide update PR raised: #893.
Thanks for the help @jodh-intel; it was confusing to see a directory pkg that is not a vendor dir; that's why..
And I have good news and bad news:
The good news is that with the latest (master) the error occurs but no longer completely crashes the runtime. It just says "message too long" (not a huge issue)_
The bad news is that simply uncommenting the length check doesn't solve the problem, in fact. if I uncomment that line (and recompile), the system will crash and halt everything (again).
Hi @dhrp - yes, there is more to this than we initially thought. In fact, the length check needs to be removed from the runtime, the agent and the proxy. But an env var > 2995
bytes kills the shim fwics:
$ sudo apt-get -y install utfout
$ export LC_ALL=C
$ docker run -e FOO=$(utfout a -r 2994) -ti busybox true
$ docker run -e FOO=$(utfout a -r 2995) -ti busybox true
handle_proxy_response:616:Error response received from proxy at /run/virtcontainers/pods/2b2136e05eff540ff854d76448b512132b8027132b3be77f19f0d4612c08985c/proxy.sock: {"msg":"vm: unknown token 4IgNPn14pYSZXjTQtgAnXtIQ2YHvuU5nU5bvZQkQq0s="}
/usr/libexec/clear-containers/cc-shim: Shim received an error in responseto ConnectShim command, exiting
Could you take a look @amshinde?
re-ping @amshinde :)
Description of problem
When I try to start a container with a (very) long docker run command the runtime errors with something like "message too long"
The real issue is that the runtime then also immediately stops working and I can no longer 'docker ps' or start or stop containers on this host until I reboot.
Steps to reproduce:
$ docker run -e ENV11000=$ENV11000 ubuntu echo $ENV11000
where $ENV11000 is a string more then 10240 characters long.I have traced the source of the "message too long" to here: https://github.com/containers/virtcontainers/blob/54b8c0cc68933561bc947c8b76aa491bb925f3ae/pkg/hyperstart/hyperstart.go#L345
journalctl thows some errors like:
Expected result
the container runs
Actual result
see above
root@c1:~# cc-collect-data.sh
Meta details
Running
cc-collect-data.sh
version3.0.10 (commit 3d402d1)
at2018-01-04.08:59:52.838419177
.Runtime is
/usr/bin/cc-runtime
.cc-env
Output of "
/usr/bin/cc-runtime cc-env
":Runtime config files
Runtime default config files
Runtime config file contents
Config file
/etc/clear-containers/configuration.toml
not found Output of "cat "/usr/share/defaults/clear-containers/configuration.toml"
":Logfiles
Runtime logs
Recent runtime problems found in system journal:
Proxy logs
No recent proxy problems found in system journal.
Shim logs
No recent shim problems found in system journal.
Container manager details
Have
docker
Docker
Output of "
docker info
":No
kubectl
Packages
Have
dpkg
Output of "dpkg -l|egrep "(cc-proxy|cc-runtime|cc-shim|clear-containers-image|linux-container|qemu-lite|qemu-system-x86|cc-oci-runtime)"
":No
rpm