Closed rajeshP524 closed 1 month ago
[root@f27-h32-000-r630 ~]# crucible repo info
*** NOTICE: Crucible is at least 14 commits behind. See 'crucible repo info' for details. ***
Project Name Project Type Git Local Branch Git Remote URL Git Remote Branch Git Remote Name Local Changes Current Commit Updates
crucible master https://github.com/perftool-incubator/crucible.git master origin N d956969 -
hwlatdetect benchmarks main https://github.com/perftool-incubator/bench-hwlatdetect main origin N 9e15e72 -
cyclictest benchmarks main https://github.com/perftool-incubator/bench-cyclictest main origin N a99e81e -
uperf benchmarks master https://github.com/perftool-incubator/bench-uperf master origin N d954df7 -
tracer benchmarks main https://github.com/perftool-incubator/bench-tracer main origin N d3bc133 -
trafficgen benchmarks main https://github.com/perftool-incubator/bench-trafficgen main origin N ae58360 -
iperf benchmarks main https://github.com/perftool-incubator/bench-iperf main origin N 8726b24 -
oslat benchmarks master https://github.com/perftool-incubator/bench-oslat master origin N 6dc610f -
flexran benchmarks main https://github.com/perftool-incubator/bench-flexran main origin N 7e12bb9 -
fio benchmarks master https://github.com/perftool-incubator/bench-fio master origin N d31e4ff -
ovs tools master https://github.com/perftool-incubator/tool-ovs master origin N 17eaf28 -
ftrace tools master https://github.com/perftool-incubator/tool-ftrace master origin N 152d39f -
forkstat tools main https://github.com/perftool-incubator/tool-forkstat main origin N 92fe253 -
procstat tools master https://github.com/perftool-incubator/tool-procstat master origin N 05cb914 -
kernel tools master https://github.com/perftool-incubator/tool-kernel master origin N 1a4927e -
sysstat tools master https://github.com/perftool-incubator/tool-sysstat master origin N 9283a7a -
rt-trace-bpf tools main https://github.com/perftool-incubator/tool-rt-trace-bpf main origin N 30a92b6 -
multiplex core master https://github.com/perftool-incubator/multiplex master origin N 4cc955f -
crucible-ci core main https://github.com/perftool-incubator/crucible-ci main origin N 9c03634 -
packrat core master https://github.com/perftool-incubator/packrat master origin N 4fac037 -
workshop core master https://github.com/perftool-incubator/workshop master origin N ed5eca9 -
toolbox core main https://github.com/perftool-incubator/toolbox main origin N f23f81a -
roadblock core master https://github.com/perftool-incubator/roadblock master origin Y d1f346d -
CommonDataModel core master https://github.com/perftool-incubator/CommonDataModel master origin N 1fc4257 -
rickshaw core master https://github.com/perftool-incubator/rickshaw master origin Y 53149c1 14
examples docs main https://github.com/perftool-incubator/crucible-examples main origin N 4adedd7 -
testing docs master https://github.com/perftool-incubator/testing-repo master origin N 47733b6 -
#!/bin/bash
provider_1_server_1="192.18.1.149"
provider_1_client_1="192.18.1.143"
provider_1_server_2="192.18.1.36"
provider_1_client_2="192.18.1.146"
provider_1_server_3="192.18.1.240"
provider_1_client_3="192.18.1.42"
provider_1_server_4="192.18.1.114"
provider_1_client_4="192.18.1.245"
provider_1_server_5="192.18.1.31"
provider_1_client_5="192.18.1.103"
provider_1_server_6="192.18.1.167"
provider_1_client_6="192.18.1.69"
provider_1_server_7="192.18.1.117"
provider_1_client_7="192.18.1.166"
provider_1_server_8="192.18.1.5"
provider_1_client_8="192.18.1.185"
provider_1_server_9="192.18.1.53"
provider_1_client_9="192.18.1.249"
provider_1_server_10="192.18.1.57"
provider_1_client_10="192.18.1.181"
provider_2_server_1="192.18.2.130"
provider_2_client_1="192.18.2.165"
provider_2_server_2="192.18.2.216"
provider_2_client_2="192.18.2.3"
provider_2_server_3="192.18.2.60"
provider_2_client_3="192.18.2.202"
provider_2_server_4="192.18.2.54"
provider_2_client_4="192.18.2.142"
provider_2_server_5="192.18.2.123"
provider_2_client_5="192.18.2.39"
provider_2_server_6="192.18.2.87"
provider_2_client_6="192.18.2.248"
provider_2_server_7="192.18.2.194"
provider_2_client_7="192.18.2.126"
provider_2_server_8="192.18.2.175"
provider_2_client_8="192.18.2.84"
provider_2_server_9="192.18.2.254"
provider_2_client_9="192.18.2.242"
provider_2_server_10="192.18.2.91"
provider_2_client_10="192.18.2.217"
provider_3_server_1="192.18.3.16"
provider_3_client_1="192.18.3.7"
provider_3_server_2="192.18.3.22"
provider_3_client_2="192.18.3.85"
provider_3_server_3="192.18.3.237"
provider_3_client_3="192.18.3.81"
provider_3_server_4="192.18.3.217"
provider_3_client_4="192.18.3.63"
provider_3_server_5="192.18.3.132"
provider_3_client_5="192.18.3.137"
provider_3_server_6="192.18.3.164"
provider_3_client_6="192.18.3.24"
provider_3_server_7="192.18.3.74"
provider_3_client_7="192.18.3.247"
provider_3_server_8="192.18.3.112"
provider_3_client_8="192.18.3.109"
provider_3_server_9="192.18.3.50"
provider_3_client_9="192.18.3.61"
provider_3_server_10="192.18.3.133"
provider_3_client_10="192.18.3.37"
provider_4_server_1="192.18.4.213"
provider_4_client_1="192.18.4.183"
provider_4_server_2="192.18.4.151"
provider_4_client_2="192.18.4.184"
provider_4_server_3="192.18.4.221"
provider_4_client_3="192.18.4.57"
provider_4_server_4="192.18.4.156"
provider_4_client_4="192.18.4.203"
provider_4_server_5="192.18.4.126"
provider_4_client_5="192.18.4.3"
provider_4_server_6="192.18.4.62"
provider_4_client_6="192.18.4.80"
provider_4_server_7="192.18.4.252"
provider_4_client_7="192.18.4.135"
provider_4_server_8="192.18.4.247"
provider_4_client_8="192.18.4.208"
provider_4_server_9="192.18.4.44"
provider_4_client_9="192.18.4.52"
provider_4_server_10="192.18.4.248"
provider_4_client_10="192.18.4.194"
provider_5_server_1="192.18.5.123"
provider_5_client_1="192.18.5.152"
provider_5_server_2="192.18.5.210"
provider_5_client_2="192.18.5.61"
provider_5_server_3="192.18.5.65"
provider_5_client_3="192.18.5.222"
provider_5_server_4="192.18.5.70"
provider_5_client_4="192.18.5.10"
provider_5_server_5="192.18.5.44"
provider_5_client_5="192.18.5.21"
provider_5_server_6="192.18.5.20"
provider_5_client_6="192.18.5.80"
provider_5_server_7="192.18.5.79"
provider_5_client_7="192.18.5.115"
provider_5_server_8="192.18.5.156"
provider_5_client_8="192.18.5.108"
provider_5_server_9="192.18.5.41"
provider_5_client_9="192.18.5.143"
provider_5_server_10="192.18.5.121"
provider_5_client_10="192.18.5.224"
provider_6_server_1="192.18.6.208"
provider_6_client_1="192.18.6.108"
provider_6_server_2="192.18.6.38"
provider_6_client_2="192.18.6.148"
provider_6_server_3="192.18.6.187"
provider_6_client_3="192.18.6.53"
provider_6_server_4="192.18.6.50"
provider_6_client_4="192.18.6.30"
provider_6_server_5="192.18.6.68"
provider_6_client_5="192.18.6.146"
provider_6_server_6="192.18.6.33"
provider_6_client_6="192.18.6.75"
provider_6_server_7="192.18.6.246"
provider_6_client_7="192.18.6.231"
provider_6_server_8="192.18.6.250"
provider_6_client_8="192.18.6.96"
provider_6_server_9="192.18.6.64"
provider_6_client_9="192.18.6.47"
provider_6_server_10="192.18.6.46"
provider_6_client_10="192.18.6.182"
provider_7_server_1="192.18.7.112"
provider_7_client_1="192.18.7.174"
provider_7_server_2="192.18.7.116"
provider_7_client_2="192.18.7.75"
provider_7_server_3="192.18.7.124"
provider_7_client_3="192.18.7.98"
provider_7_server_4="192.18.7.38"
provider_7_client_4="192.18.7.115"
provider_7_server_5="192.18.7.237"
provider_7_client_5="192.18.7.156"
provider_7_server_6="192.18.7.129"
provider_7_client_6="192.18.7.181"
provider_7_server_7="192.18.7.206"
provider_7_client_7="192.18.7.139"
provider_7_server_8="192.18.7.111"
provider_7_client_8="192.18.7.6"
provider_7_server_9="192.18.7.182"
provider_7_client_9="192.18.7.61"
provider_7_server_10="192.18.7.37"
provider_7_client_10="192.18.7.86"
provider_8_server_1="192.18.8.151"
provider_8_client_1="192.18.8.194"
provider_8_server_2="192.18.8.89"
provider_8_client_2="192.18.8.28"
provider_8_server_3="192.18.8.98"
provider_8_client_3="192.18.8.13"
provider_8_server_4="192.18.8.154"
provider_8_client_4="192.18.8.168"
provider_8_server_5="192.18.8.72"
provider_8_client_5="192.18.8.133"
provider_8_server_6="192.18.8.226"
provider_8_client_6="192.18.8.205"
provider_8_server_7="192.18.8.211"
provider_8_client_7="192.18.8.77"
provider_8_server_8="192.18.8.76"
provider_8_client_8="192.18.8.140"
provider_8_server_9="192.18.8.179"
provider_8_client_9="192.18.8.229"
provider_8_server_10="192.18.8.104"
provider_8_client_10="192.18.8.245"
provider_9_server_1="192.18.9.165"
provider_9_client_1="192.18.9.39"
provider_9_server_2="192.18.9.148"
provider_9_client_2="192.18.9.246"
provider_9_server_3="192.18.9.220"
provider_9_client_3="192.18.9.86"
provider_9_server_4="192.18.9.14"
provider_9_client_4="192.18.9.62"
provider_9_server_5="192.18.9.114"
provider_9_client_5="192.18.9.137"
provider_9_server_6="192.18.9.47"
provider_9_client_6="192.18.9.69"
provider_9_server_7="192.18.9.215"
provider_9_client_7="192.18.9.175"
provider_9_server_8="192.18.9.176"
provider_9_client_8="192.18.9.162"
provider_9_server_9="192.18.9.103"
provider_9_client_9="192.18.9.51"
provider_9_server_10="192.18.9.36"
provider_9_client_10="192.18.9.228"
provider_10_server_1="192.18.10.10"
provider_10_client_1="192.18.10.203"
provider_10_server_2="192.18.10.7"
provider_10_client_2="192.18.10.142"
provider_10_server_3="192.18.10.120"
provider_10_client_3="192.18.10.59"
provider_10_server_4="192.18.10.57"
provider_10_client_4="192.18.10.161"
provider_10_server_5="192.18.10.201"
provider_10_client_5="192.18.10.251"
provider_10_server_6="192.18.10.77"
provider_10_client_6="192.18.10.72"
provider_10_server_7="192.18.10.62"
provider_10_client_7="192.18.10.173"
provider_10_server_8="192.18.10.165"
provider_10_client_8="192.18.10.190"
provider_10_server_9="192.18.10.164"
provider_10_client_9="192.18.10.73"
provider_10_server_10="192.18.10.217"
provider_10_client_10="192.18.10.20"
provider_11_server_1="192.18.11.119"
provider_11_client_1="192.18.11.61"
provider_11_server_2="192.18.11.55"
provider_11_client_2="192.18.11.81"
provider_11_server_3="192.18.11.39"
provider_11_client_3="192.18.11.4"
provider_11_server_4="192.18.11.242"
provider_11_client_4="192.18.11.133"
provider_11_server_5="192.18.11.41"
provider_11_client_5="192.18.11.182"
provider_11_server_6="192.18.11.50"
provider_11_client_6="192.18.11.185"
provider_11_server_7="192.18.11.175"
provider_11_client_7="192.18.11.215"
provider_11_server_8="192.18.11.194"
provider_11_client_8="192.18.11.211"
provider_11_server_9="192.18.11.82"
provider_11_client_9="192.18.11.47"
provider_11_server_10="192.18.11.68"
provider_11_client_10="192.18.11.239"
provider_12_server_1="192.18.12.196"
provider_12_client_1="192.18.12.119"
provider_12_server_2="192.18.12.55"
provider_12_client_2="192.18.12.5"
provider_12_server_3="192.18.12.124"
provider_12_client_3="192.18.12.31"
provider_12_server_4="192.18.12.216"
provider_12_client_4="192.18.12.107"
provider_12_server_5="192.18.12.54"
provider_12_client_5="192.18.12.94"
provider_12_server_6="192.18.12.100"
provider_12_client_6="192.18.12.111"
provider_12_server_7="192.18.12.165"
provider_12_client_7="192.18.12.174"
provider_12_server_8="192.18.12.72"
provider_12_client_8="192.18.12.128"
provider_12_server_9="192.18.12.98"
provider_12_client_9="192.18.12.126"
provider_12_server_10="192.18.12.159"
provider_12_client_10="192.18.12.8"
provider_13_server_1="192.18.13.8"
provider_13_client_1="192.18.13.57"
provider_13_server_2="192.18.13.198"
provider_13_client_2="192.18.13.34"
provider_13_server_3="192.18.13.40"
provider_13_client_3="192.18.13.79"
provider_13_server_4="192.18.13.45"
provider_13_client_4="192.18.13.93"
provider_13_server_5="192.18.13.154"
provider_13_client_5="192.18.13.240"
provider_13_server_6="192.18.13.156"
provider_13_client_6="192.18.13.56"
provider_13_server_7="192.18.13.194"
provider_13_client_7="192.18.13.120"
provider_13_server_8="192.18.13.102"
provider_13_client_8="192.18.13.106"
provider_13_server_9="192.18.13.126"
provider_13_client_9="192.18.13.28"
provider_13_server_10="192.18.13.112"
provider_13_client_10="192.18.13.73"
provider_14_server_1="192.18.14.62"
provider_14_client_1="192.18.14.121"
provider_14_server_2="192.18.14.26"
provider_14_client_2="192.18.14.24"
provider_14_server_3="192.18.14.82"
provider_14_client_3="192.18.14.66"
provider_14_server_4="192.18.14.52"
provider_14_client_4="192.18.14.148"
provider_14_server_5="192.18.14.112"
provider_14_client_5="192.18.14.125"
provider_14_server_6="192.18.14.104"
provider_14_client_6="192.18.14.77"
provider_14_server_7="192.18.14.234"
provider_14_client_7="192.18.14.174"
provider_14_server_8="192.18.14.157"
provider_14_client_8="192.18.14.144"
provider_14_server_9="192.18.14.122"
provider_14_client_9="192.18.14.90"
provider_14_server_10="192.18.14.231"
provider_14_client_10="192.18.14.38"
provider_15_server_1="192.18.15.83"
provider_15_client_1="192.18.15.216"
provider_15_server_2="192.18.15.120"
provider_15_client_2="192.18.15.112"
provider_15_server_3="192.18.15.3"
provider_15_client_3="192.18.15.176"
provider_15_server_4="192.18.15.139"
provider_15_client_4="192.18.15.225"
provider_15_server_5="192.18.15.230"
provider_15_client_5="192.18.15.109"
provider_15_server_6="192.18.15.182"
provider_15_client_6="192.18.15.31"
provider_15_server_7="192.18.15.102"
provider_15_client_7="192.18.15.199"
provider_15_server_8="192.18.15.150"
provider_15_client_8="192.18.15.95"
provider_15_server_9="192.18.15.43"
provider_15_client_9="192.18.15.85"
provider_15_server_10="192.18.15.205"
provider_15_client_10="192.18.15.246"
provider_16_server_1="192.18.16.140"
provider_16_client_1="192.18.16.165"
provider_16_server_2="192.18.16.196"
provider_16_client_2="192.18.16.248"
provider_16_server_3="192.18.16.55"
provider_16_client_3="192.18.16.103"
provider_16_server_4="192.18.16.113"
provider_16_client_4="192.18.16.15"
provider_16_server_5="192.18.16.136"
provider_16_client_5="192.18.16.213"
provider_16_server_6="192.18.16.123"
provider_16_client_6="192.18.16.205"
provider_16_server_7="192.18.16.109"
provider_16_client_7="192.18.16.127"
provider_16_server_8="192.18.16.66"
provider_16_client_8="192.18.16.235"
provider_16_server_9="192.18.16.47"
provider_16_client_9="192.18.16.44"
provider_16_server_10="192.18.16.102"
provider_16_client_10="192.18.16.198"
provider_17_server_1="192.18.17.164"
provider_17_client_1="192.18.17.182"
provider_17_server_2="192.18.17.136"
provider_17_client_2="192.18.17.156"
provider_17_server_3="192.18.17.85"
provider_17_client_3="192.18.17.100"
provider_17_server_4="192.18.17.127"
provider_17_client_4="192.18.17.163"
provider_17_server_5="192.18.17.161"
provider_17_client_5="192.18.17.91"
provider_17_server_6="192.18.17.175"
provider_17_client_6="192.18.17.57"
provider_17_server_7="192.18.17.207"
provider_17_client_7="192.18.17.214"
provider_17_server_8="192.18.17.211"
provider_17_client_8="192.18.17.9"
provider_17_server_9="192.18.17.86"
provider_17_client_9="192.18.17.71"
provider_17_server_10="192.18.17.61"
provider_17_client_10="192.18.17.213"
provider_18_server_1="192.18.18.57"
provider_18_client_1="192.18.18.52"
provider_18_server_2="192.18.18.208"
provider_18_client_2="192.18.18.198"
provider_18_server_3="192.18.18.238"
provider_18_client_3="192.18.18.47"
provider_18_server_4="192.18.18.119"
provider_18_client_4="192.18.18.28"
provider_18_server_5="192.18.18.89"
provider_18_client_5="192.18.18.96"
provider_18_server_6="192.18.18.192"
provider_18_client_6="192.18.18.20"
provider_18_server_7="192.18.18.35"
provider_18_client_7="192.18.18.158"
provider_18_server_8="192.18.18.55"
provider_18_client_8="192.18.18.210"
provider_18_server_9="192.18.18.83"
provider_18_client_9="192.18.18.127"
provider_18_server_10="192.18.18.68"
provider_18_client_10="192.18.18.46"
provider_19_server_1="192.18.19.97"
provider_19_client_1="192.18.19.24"
provider_19_server_2="192.18.19.70"
provider_19_client_2="192.18.19.249"
provider_19_server_3="192.18.19.161"
provider_19_client_3="192.18.19.143"
provider_19_server_4="192.18.19.226"
provider_19_client_4="192.18.19.140"
provider_19_server_5="192.18.19.173"
provider_19_client_5="192.18.19.191"
provider_19_server_6="192.18.19.137"
provider_19_client_6="192.18.19.109"
provider_19_server_7="192.18.19.35"
provider_19_client_7="192.18.19.208"
provider_19_server_8="192.18.19.201"
provider_19_client_8="192.18.19.159"
provider_19_server_9="192.18.19.223"
provider_19_client_9="192.18.19.62"
provider_19_server_10="192.18.19.177"
provider_19_client_10="192.18.19.98"
provider_20_server_1="192.18.20.189"
provider_20_client_1="192.18.20.208"
provider_20_server_2="192.18.20.175"
provider_20_client_2="192.18.20.126"
provider_20_server_3="192.18.20.236"
provider_20_client_3="192.18.20.36"
provider_20_server_4="192.18.20.139"
provider_20_client_4="192.18.20.30"
provider_20_server_5="192.18.20.137"
provider_20_client_5="192.18.20.181"
provider_20_server_6="192.18.20.26"
provider_20_client_6="192.18.20.61"
provider_20_server_7="192.18.20.159"
provider_20_client_7="192.18.20.74"
provider_20_server_8="192.18.20.3"
provider_20_client_8="192.18.20.8"
provider_20_server_9="192.18.20.140"
provider_20_client_9="192.18.20.84"
provider_20_server_10="192.18.20.83"
provider_20_client_10="192.18.20.150"
client_userenv="stream8"
tags="browbeat:browbeat"
profile="mv_params.json"
num_samples=3
time crucible run uperf \
--test-order r \
--mv-params $profile \
--tags "${tags}" \
--num-samples $num_samples \
--max-sample-failures 1 \
--endpoint remotehost,user:root,host:$provider_1_client_1,client:1-1,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_server_1,server:1-1,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_client_2,client:2-2,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_server_2,server:2-2,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_client_3,client:3-3,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_server_3,server:3-3,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_client_4,client:4-4,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_server_4,server:4-4,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_client_5,client:5-5,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_server_5,server:5-5,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_client_6,client:6-6,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_server_6,server:6-6,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_client_7,client:7-7,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_server_7,server:7-7,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_client_8,client:8-8,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_server_8,server:8-8,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_client_9,client:9-9,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_server_9,server:9-9,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_client_10,client:10-10,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_1_server_10,server:10-10,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_client_1,client:11-11,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_server_1,server:11-11,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_client_2,client:12-12,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_server_2,server:12-12,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_client_3,client:13-13,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_server_3,server:13-13,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_client_4,client:14-14,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_server_4,server:14-14,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_client_5,client:15-15,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_server_5,server:15-15,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_client_6,client:16-16,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_server_6,server:16-16,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_client_7,client:17-17,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_server_7,server:17-17,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_client_8,client:18-18,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_server_8,server:18-18,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_client_9,client:19-19,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_server_9,server:19-19,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_client_10,client:20-20,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_2_server_10,server:20-20,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_client_1,client:21-21,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_server_1,server:21-21,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_client_2,client:22-22,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_server_2,server:22-22,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_client_3,client:23-23,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_server_3,server:23-23,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_client_4,client:24-24,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_server_4,server:24-24,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_client_5,client:25-25,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_server_5,server:25-25,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_client_6,client:26-26,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_server_6,server:26-26,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_client_7,client:27-27,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_server_7,server:27-27,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_client_8,client:28-28,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_server_8,server:28-28,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_client_9,client:29-29,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_server_9,server:29-29,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_client_10,client:30-30,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_3_server_10,server:30-30,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_client_1,client:31-31,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_server_1,server:31-31,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_client_2,client:32-32,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_server_2,server:32-32,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_client_3,client:33-33,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_server_3,server:33-33,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_client_4,client:34-34,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_server_4,server:34-34,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_client_5,client:35-35,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_server_5,server:35-35,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_client_6,client:36-36,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_server_6,server:36-36,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_client_7,client:37-37,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_server_7,server:37-37,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_client_8,client:38-38,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_server_8,server:38-38,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_client_9,client:39-39,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_server_9,server:39-39,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_client_10,client:40-40,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_4_server_10,server:40-40,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_client_1,client:41-41,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_server_1,server:41-41,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_client_2,client:42-42,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_server_2,server:42-42,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_client_3,client:43-43,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_server_3,server:43-43,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_client_4,client:44-44,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_server_4,server:44-44,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_client_5,client:45-45,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_server_5,server:45-45,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_client_6,client:46-46,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_server_6,server:46-46,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_client_7,client:47-47,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_server_7,server:47-47,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_client_8,client:48-48,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_server_8,server:48-48,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_client_9,client:49-49,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_server_9,server:49-49,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_client_10,client:50-50,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_5_server_10,server:50-50,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_client_1,client:51-51,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_server_1,server:51-51,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_client_2,client:52-52,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_server_2,server:52-52,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_client_3,client:53-53,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_server_3,server:53-53,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_client_4,client:54-54,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_server_4,server:54-54,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_client_5,client:55-55,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_server_5,server:55-55,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_client_6,client:56-56,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_server_6,server:56-56,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_client_7,client:57-57,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_server_7,server:57-57,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_client_8,client:58-58,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_server_8,server:58-58,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_client_9,client:59-59,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_server_9,server:59-59,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_client_10,client:60-60,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_6_server_10,server:60-60,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_client_1,client:61-61,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_server_1,server:61-61,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_client_2,client:62-62,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_server_2,server:62-62,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_client_3,client:63-63,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_server_3,server:63-63,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_client_4,client:64-64,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_server_4,server:64-64,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_client_5,client:65-65,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_server_5,server:65-65,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_client_6,client:66-66,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_server_6,server:66-66,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_client_7,client:67-67,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_server_7,server:67-67,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_client_8,client:68-68,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_server_8,server:68-68,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_client_9,client:69-69,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_server_9,server:69-69,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_client_10,client:70-70,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_7_server_10,server:70-70,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_client_1,client:71-71,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_server_1,server:71-71,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_client_2,client:72-72,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_server_2,server:72-72,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_client_3,client:73-73,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_server_3,server:73-73,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_client_4,client:74-74,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_server_4,server:74-74,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_client_5,client:75-75,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_server_5,server:75-75,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_client_6,client:76-76,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_server_6,server:76-76,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_client_7,client:77-77,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_server_7,server:77-77,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_client_8,client:78-78,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_server_8,server:78-78,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_client_9,client:79-79,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_server_9,server:79-79,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_client_10,client:80-80,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_8_server_10,server:80-80,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_client_1,client:81-81,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_server_1,server:81-81,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_client_2,client:82-82,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_server_2,server:82-82,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_client_3,client:83-83,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_server_3,server:83-83,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_client_4,client:84-84,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_server_4,server:84-84,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_client_5,client:85-85,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_server_5,server:85-85,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_client_6,client:86-86,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_server_6,server:86-86,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_client_7,client:87-87,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_server_7,server:87-87,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_client_8,client:88-88,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_server_8,server:88-88,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_client_9,client:89-89,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_server_9,server:89-89,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_client_10,client:90-90,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_9_server_10,server:90-90,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_client_1,client:91-91,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_server_1,server:91-91,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_client_2,client:92-92,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_server_2,server:92-92,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_client_3,client:93-93,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_server_3,server:93-93,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_client_4,client:94-94,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_server_4,server:94-94,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_client_5,client:95-95,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_server_5,server:95-95,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_client_6,client:96-96,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_server_6,server:96-96,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_client_7,client:97-97,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_server_7,server:97-97,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_client_8,client:98-98,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_server_8,server:98-98,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_client_9,client:99-99,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_server_9,server:99-99,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_client_10,client:100-100,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_10_server_10,server:100-100,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_client_1,client:101-101,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_server_1,server:101-101,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_client_2,client:102-102,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_server_2,server:102-102,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_client_3,client:103-103,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_server_3,server:103-103,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_client_4,client:104-104,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_server_4,server:104-104,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_client_5,client:105-105,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_server_5,server:105-105,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_client_6,client:106-106,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_server_6,server:106-106,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_client_7,client:107-107,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_server_7,server:107-107,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_client_8,client:108-108,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_server_8,server:108-108,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_client_9,client:109-109,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_server_9,server:109-109,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_client_10,client:110-110,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_11_server_10,server:110-110,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_client_1,client:111-111,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_server_1,server:111-111,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_client_2,client:112-112,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_server_2,server:112-112,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_client_3,client:113-113,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_server_3,server:113-113,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_client_4,client:114-114,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_server_4,server:114-114,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_client_5,client:115-115,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_server_5,server:115-115,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_client_6,client:116-116,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_server_6,server:116-116,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_client_7,client:117-117,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_server_7,server:117-117,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_client_8,client:118-118,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_server_8,server:118-118,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_client_9,client:119-119,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_server_9,server:119-119,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_client_10,client:120-120,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_12_server_10,server:120-120,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_client_1,client:121-121,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_server_1,server:121-121,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_client_2,client:122-122,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_server_2,server:122-122,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_client_3,client:123-123,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_server_3,server:123-123,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_client_4,client:124-124,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_server_4,server:124-124,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_client_5,client:125-125,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_server_5,server:125-125,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_client_6,client:126-126,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_server_6,server:126-126,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_client_7,client:127-127,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_server_7,server:127-127,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_client_8,client:128-128,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_server_8,server:128-128,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_client_9,client:129-129,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_server_9,server:129-129,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_client_10,client:130-130,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_13_server_10,server:130-130,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_client_1,client:131-131,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_server_1,server:131-131,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_client_2,client:132-132,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_server_2,server:132-132,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_client_3,client:133-133,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_server_3,server:133-133,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_client_4,client:134-134,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_server_4,server:134-134,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_client_5,client:135-135,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_server_5,server:135-135,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_client_6,client:136-136,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_server_6,server:136-136,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_client_7,client:137-137,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_server_7,server:137-137,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_client_8,client:138-138,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_server_8,server:138-138,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_client_9,client:139-139,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_server_9,server:139-139,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_client_10,client:140-140,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_14_server_10,server:140-140,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_client_1,client:141-141,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_server_1,server:141-141,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_client_2,client:142-142,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_server_2,server:142-142,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_client_3,client:143-143,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_server_3,server:143-143,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_client_4,client:144-144,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_server_4,server:144-144,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_client_5,client:145-145,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_server_5,server:145-145,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_client_6,client:146-146,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_server_6,server:146-146,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_client_7,client:147-147,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_server_7,server:147-147,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_client_8,client:148-148,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_server_8,server:148-148,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_client_9,client:149-149,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_server_9,server:149-149,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_client_10,client:150-150,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_15_server_10,server:150-150,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_client_1,client:151-151,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_server_1,server:151-151,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_client_2,client:152-152,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_server_2,server:152-152,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_client_3,client:153-153,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_server_3,server:153-153,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_client_4,client:154-154,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_server_4,server:154-154,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_client_5,client:155-155,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_server_5,server:155-155,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_client_6,client:156-156,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_server_6,server:156-156,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_client_7,client:157-157,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_server_7,server:157-157,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_client_8,client:158-158,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_server_8,server:158-158,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_client_9,client:159-159,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_server_9,server:159-159,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_client_10,client:160-160,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_16_server_10,server:160-160,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_client_1,client:161-161,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_server_1,server:161-161,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_client_2,client:162-162,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_server_2,server:162-162,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_client_3,client:163-163,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_server_3,server:163-163,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_client_4,client:164-164,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_server_4,server:164-164,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_client_5,client:165-165,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_server_5,server:165-165,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_client_6,client:166-166,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_server_6,server:166-166,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_client_7,client:167-167,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_server_7,server:167-167,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_client_8,client:168-168,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_server_8,server:168-168,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_client_9,client:169-169,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_server_9,server:169-169,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_client_10,client:170-170,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_17_server_10,server:170-170,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_client_1,client:171-171,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_server_1,server:171-171,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_client_2,client:172-172,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_server_2,server:172-172,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_client_3,client:173-173,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_server_3,server:173-173,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_client_4,client:174-174,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_server_4,server:174-174,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_client_5,client:175-175,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_server_5,server:175-175,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_client_6,client:176-176,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_server_6,server:176-176,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_client_7,client:177-177,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_server_7,server:177-177,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_client_8,client:178-178,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_server_8,server:178-178,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_client_9,client:179-179,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_server_9,server:179-179,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_client_10,client:180-180,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_18_server_10,server:180-180,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_client_1,client:181-181,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_server_1,server:181-181,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_client_2,client:182-182,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_server_2,server:182-182,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_client_3,client:183-183,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_server_3,server:183-183,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_client_4,client:184-184,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_server_4,server:184-184,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_client_5,client:185-185,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_server_5,server:185-185,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_client_6,client:186-186,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_server_6,server:186-186,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_client_7,client:187-187,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_server_7,server:187-187,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_client_8,client:188-188,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_server_8,server:188-188,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_client_9,client:189-189,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_server_9,server:189-189,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_client_10,client:190-190,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_19_server_10,server:190-190,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_client_1,client:191-191,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_server_1,server:191-191,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_client_2,client:192-192,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_server_2,server:192-192,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_client_3,client:193-193,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_server_3,server:193-193,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_client_4,client:194-194,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_server_4,server:194-194,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_client_5,client:195-195,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_server_5,server:195-195,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_client_6,client:196-196,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_server_6,server:196-196,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_client_7,client:197-197,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_server_7,server:197-197,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_client_8,client:198-198,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_server_8,server:198-198,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_client_9,client:199-199,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_server_9,server:199-199,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_client_10,client:200-200,userenv:${client_userenv} \
--endpoint remotehost,user:root,host:$provider_20_server_10,server:200-200,userenv:${client_userenv}
At first glance it appears that the big difference between what is being run here and the scalability work that we previously did is the way that endpoints are being used.
In the previous scalability work, the emphasis was on getting as many engines running as possible and stress testing the various things that relate to that (https://github.com/perftool-incubator/roadblock code, individual roadblock invocations -- ie. timeouts, etc.). In that work we ran as many as 10,000 separate engines and as many as 40 endpoints using 10 baremetal hosts.
The fact that we were able to synchronize 10,000+ (the engines + endpoints + controller) roadblock participants gives us fairly high confidence that roadblock is in good shape for this degree of scaling and maybe even much more.
Since there are 400 separate VMs being used here and each one is a separate endpoint, it seems logical to initially focus on endpoint scaling as the likely problem. Each endpoint is going to be a separate process on the Crucible controller that is spawned by rickshaw-run
(all inside a single podman container) so the stress points are different than what we had previously focused on.
A major piece of the work to address this issue is taking place here: https://github.com/perftool-incubator/rickshaw/pull/483
We are using
remotehost
endpoint in crucible for benchmarking a set of openstack VMs distributed across different provider networks. Initially we started with a small number of VMs and then started increasing the VM count which acts as servers and clients. Until 80 VMs(40 servers and 40 clients) we did not see any concerns. And after that we tried out with 400 VMs(200 servers and 200 clients). In this we saw that the test was carried out well.. but in the post processing we did hit errors like below.[2024-02-09 07:06:29.184][STDOUT] sh: fork: retry: Resource temporarily unavailable [2024-02-09 07:06:29.187][STDOUT] sh: fork: retry: Resource temporarily unavailable
Since crucible services are running as containers.. there are some PID limits associated with it. As a workaround to the above error.. we did increase the PID limits for the containers.Later after that.. we have tried out with 1400 VMs(700 servers and 700 clients) to understand the crucible controller limits. We could not go forward because of the
too many open files limit error
on crucible controller.Deploying endpoints endpoint-deploy-timeout adjusted to 190440 seconds engine-script-timeout adjusted to 190440 seconds Can't exec "/bin/sh": Too many open files at /opt/crucible/subprojects/core/rickshaw/rickshaw-run line 2170. Can't exec "/bin/sh": Too many open files at /opt/crucible/subprojects/core/rickshaw/rickshaw-run line 2170.
After some time... Roadblock timed out and the test failed. We have tried increasing open file limits on the crucible controller and still it seems that crucible is opening too many files or there is some limitation for the containers.