k-rister
commented
6 months ago [root@f27-h32-000-r630 ~]# crucible repo info
*** NOTICE: Crucible is at least 14 commits behind. See 'crucible repo info' for details. ***
Project Name Project Type Git Local Branch Git Remote URL Git Remote Branch Git Remote Name Local Changes Current Commit Updates
crucible master https://github.com/perftool-incubator/crucible.git master origin N d956969 -
hwlatdetect benchmarks main https://github.com/perftool-incubator/bench-hwlatdetect main origin N 9e15e72 -
cyclictest benchmarks main https://github.com/perftool-incubator/bench-cyclictest main origin N a99e81e -
uperf benchmarks master https://github.com/perftool-incubator/bench-uperf master origin N d954df7 -
tracer benchmarks main https://github.com/perftool-incubator/bench-tracer main origin N d3bc133 -
trafficgen benchmarks main https://github.com/perftool-incubator/bench-trafficgen main origin N ae58360 -
iperf benchmarks main https://github.com/perftool-incubator/bench-iperf main origin N 8726b24 -
oslat benchmarks master https://github.com/perftool-incubator/bench-oslat master origin N 6dc610f -
flexran benchmarks main https://github.com/perftool-incubator/bench-flexran main origin N 7e12bb9 -
fio benchmarks master https://github.com/perftool-incubator/bench-fio master origin N d31e4ff -
ovs tools master https://github.com/perftool-incubator/tool-ovs master origin N 17eaf28 -
ftrace tools master https://github.com/perftool-incubator/tool-ftrace master origin N 152d39f -
forkstat tools main https://github.com/perftool-incubator/tool-forkstat main origin N 92fe253 -
procstat tools master https://github.com/perftool-incubator/tool-procstat master origin N 05cb914 -
kernel tools master https://github.com/perftool-incubator/tool-kernel master origin N 1a4927e -
sysstat tools master https://github.com/perftool-incubator/tool-sysstat master origin N 9283a7a -
rt-trace-bpf tools main https://github.com/perftool-incubator/tool-rt-trace-bpf main origin N 30a92b6 -
multiplex core master https://github.com/perftool-incubator/multiplex master origin N 4cc955f -
crucible-ci core main https://github.com/perftool-incubator/crucible-ci main origin N 9c03634 -
packrat core master https://github.com/perftool-incubator/packrat master origin N 4fac037 -
workshop core master https://github.com/perftool-incubator/workshop master origin N ed5eca9 -
toolbox core main https://github.com/perftool-incubator/toolbox main origin N f23f81a -
roadblock core master https://github.com/perftool-incubator/roadblock master origin Y d1f346d -
CommonDataModel core master https://github.com/perftool-incubator/CommonDataModel master origin N 1fc4257 -
rickshaw core master https://github.com/perftool-incubator/rickshaw master origin Y 53149c1 14
examples docs main https://github.com/perftool-incubator/crucible-examples main origin N 4adedd7 -
testing docs master https://github.com/perftool-incubator/testing-repo master origin N 47733b6 -
We are using
remotehostendpoint in crucible for benchmarking a set of openstack VMs distributed across different provider networks. Initially we started with a small number of VMs and then started increasing the VM count which acts as servers and clients. Until 80 VMs(40 servers and 40 clients) we did not see any concerns. And after that we tried out with 400 VMs(200 servers and 200 clients). In this we saw that the test was carried out well.. but in the post processing we did hit errors like below.[2024-02-09 07:06:29.184][STDOUT] sh: fork: retry: Resource temporarily unavailable [2024-02-09 07:06:29.187][STDOUT] sh: fork: retry: Resource temporarily unavailableSince crucible services are running as containers.. there are some PID limits associated with it. As a workaround to the above error.. we did increase the PID limits for the containers.Later after that.. we have tried out with 1400 VMs(700 servers and 700 clients) to understand the crucible controller limits. We could not go forward because of the
too many open files limit erroron crucible controller.Deploying endpoints endpoint-deploy-timeout adjusted to 190440 seconds engine-script-timeout adjusted to 190440 seconds Can't exec "/bin/sh": Too many open files at /opt/crucible/subprojects/core/rickshaw/rickshaw-run line 2170. Can't exec "/bin/sh": Too many open files at /opt/crucible/subprojects/core/rickshaw/rickshaw-run line 2170.After some time... Roadblock timed out and the test failed. We have tried increasing open file limits on the crucible controller and still it seems that crucible is opening too many files or there is some limitation for the containers.