lb: dedicated nodes placement

baabeetaa commented 1 year ago

atm each chain has a load-balancer instance. And these load-balancer instances deployed across all node within swarm cluster. It works however there is a couple of issues

docker swarm places new task on a swarm node which has least containers running. When running mixed-tasks (load-balancer/rpc..), so loads are not distributed equally on all nodes. And we have to set label for each load-balancer service to fix that ( see https://notional-labs.github.io/cosmosia/#/service_placement?id=load-balancer-service-placement )
When a chain has high traffic, the load-balancer instance eats all the bandwidth and impact other services on the same node.

So we'll move all load-balancer instances to dedicated nodes. Begin with 3 nodes (cosmosia26, cosmosia27 and cosmosia28), and could add more node later when needed. These nodes are optimized to run load-balancer instances (low cost, use little CPU, almost no diskio and high bandwidth).

baabeetaa commented 1 year ago

re-deploy all lb services

baabeetaa commented 1 year ago

[root@cosmosia26 ~]# docker ps -a
CONTAINER ID   IMAGE              COMMAND                  CREATED          STATUS          PORTS     NAMES
9bd1345f4b4f   archlinux:latest   "/bin/bash -c 'curl …"   2 minutes ago    Up 2 minutes              lb_umee.1.yp3onqdqotsqui1yb4poh8wjb
697caf164a74   archlinux:latest   "/bin/bash -c 'curl …"   2 minutes ago    Up 2 minutes              lb_stride.1.ahcrq030n96nzxb2a5yzbg87b
237a308f18fa   archlinux:latest   "/bin/bash -c 'curl …"   3 minutes ago    Up 3 minutes              lb_sifchain.1.7z73z3cq26c9a4sp1l2g9cuqn
d3872085200a   archlinux:latest   "/bin/bash -c 'curl …"   4 minutes ago    Up 4 minutes              lb_regen.1.5skkd56pomj2e2w6xa3i9xg6x
77c127f3cd16   archlinux:latest   "/bin/bash -c 'curl …"   5 minutes ago    Up 5 minutes              lb_persistent.1.s85d87clrw99vkmceestvaw8e
f47483c78c74   archlinux:latest   "/bin/bash -c 'curl …"   6 minutes ago    Up 6 minutes              lb_osmosis.1.pi6l3lyu0oxovvczyr5k6p29w
5b5e20c43489   archlinux:latest   "/bin/bash -c 'curl …"   6 minutes ago    Up 6 minutes              lb_omniflixhub.1.s1kd200081nldqac8t7lc5pi2
67087b1c1c2f   archlinux:latest   "/bin/bash -c 'curl …"   7 minutes ago    Up 7 minutes              lb_kichain.1.upwwb0ryq8w0e51is6jvzt0vk
d13bad876bf6   archlinux:latest   "/bin/bash -c 'curl …"   8 minutes ago    Up 8 minutes              lb_juno-archive.1.e05chk4557lp9gn6zwzcqbew3
653b4e0cc3ff   archlinux:latest   "/bin/bash -c 'curl …"   9 minutes ago    Up 9 minutes              lb_injective.1.e72kb4115k78cdnipoj884pat
db5fc5093c60   archlinux:latest   "/bin/bash -c 'curl …"   10 minutes ago   Up 10 minutes             lb_fetchhub.1.rbkzf1olxfh83djnbvtb0l7i9
29db4e4e8845   archlinux:latest   "/bin/bash -c 'curl …"   11 minutes ago   Up 11 minutes             lb_emoney.1.tp0l699aiplo7hr286okqb517
efa0ba20ddf5   archlinux:latest   "/bin/bash -c 'curl …"   12 minutes ago   Up 12 minutes             lb_cosmoshub-archive.1.uk7lleg2stw8ndc80n9lnlvei
205bdd8e0414   archlinux:latest   "/bin/bash -c 'curl …"   13 minutes ago   Up 13 minutes             lb_chihuahua.1.mchep0tsbqa9k3qctjm3sy7ru
81e5e5a8da97   archlinux:latest   "/bin/bash -c 'curl …"   14 minutes ago   Up 14 minutes             lb_bitcanna.1.omqtmz0hqza9wwxyq3a5dy7yi
4ce859771e39   archlinux:latest   "/bin/bash -c 'curl …"   17 minutes ago   Up 17 minutes             lb_akash.1.iea2uxbumejxijpdikr8b1s5d

[root@cosmosia27 ~]# docker ps -a
CONTAINER ID   IMAGE              COMMAND                  CREATED          STATUS          PORTS     NAMES
7e82b742db51   archlinux:latest   "/bin/bash -c 'curl …"   2 minutes ago    Up 2 minutes              lb_vidulum.1.mry5twpykylphq98oa3kvfpi5
bc1c2d99fc21   archlinux:latest   "/bin/bash -c 'curl …"   3 minutes ago    Up 3 minutes              lb_terra.1.0ofu7idjhig99wc58j2x6e381
0339cad7f447   archlinux:latest   "/bin/bash -c 'curl …"   3 minutes ago    Up 3 minutes              lb_stargaze.1.el7mdrfrany5564dwj8v5vprk
ea1857d6a373   archlinux:latest   "/bin/bash -c 'curl …"   4 minutes ago    Up 4 minutes              lb_sei.1.lx75qtgng67de30vxl9y4hxz7
d4bccb92e5d0   archlinux:latest   "/bin/bash -c 'curl …"   5 minutes ago    Up 5 minutes              lb_provenance.1.t3tdr099fw9gay65e84gzrbao
8a8e7d849235   archlinux:latest   "/bin/bash -c 'curl …"   6 minutes ago    Up 6 minutes              lb_osmosis-archive.1.qs6j3cqfyabj9z1p9elwwd9yj
db747a5354c5   archlinux:latest   "/bin/bash -c 'curl …"   7 minutes ago    Up 7 minutes              lb_mars.1.j6q2ay27swobg55e87k3avztp
d9e947e6aaa3   archlinux:latest   "/bin/bash -c 'curl …"   8 minutes ago    Up 8 minutes              lb_konstellation.1.yqwg3mzcuptstiwg9fl1xsiz9
db6dc81eb877   archlinux:latest   "/bin/bash -c 'curl …"   8 minutes ago    Up 8 minutes              lb_kava.1.kser3bjgdcxy749l365nq4f0s
ad61f8338d8f   archlinux:latest   "/bin/bash -c 'curl …"   9 minutes ago    Up 9 minutes              lb_ixo.1.jsxbmkujfbdrxm3mpoqh548la
f17e9304c17b   archlinux:latest   "/bin/bash -c 'curl …"   10 minutes ago   Up 10 minutes             lb_gravitybridge.1.oiubzvaj9ea1uopohbwta4zk1
e89ec2c29c04   archlinux:latest   "/bin/bash -c 'curl …"   11 minutes ago   Up 11 minutes             lb_evmos.1.wc1otoxbvskw4pmwifu41sffp
c2fd52bc2a7f   archlinux:latest   "/bin/bash -c 'curl …"   12 minutes ago   Up 12 minutes             lb_cyber.1.y0106w52ehey8yjmd15evkdmh
754a890d899f   archlinux:latest   "/bin/bash -c 'curl …"   13 minutes ago   Up 13 minutes             lb_cosmoshub.1.m2g4n9x8jbo361pasi3fj54qp
279aaf7671ac   archlinux:latest   "/bin/bash -c 'curl …"   15 minutes ago   Up 15 minutes             lb_bitsong.1.d8ll21elghva7yterceqzqpya
6f02e4d48413   archlinux:latest   "/bin/bash -c 'curl …"   16 minutes ago   Up 16 minutes             lb_axelar.1.ymgx9o5fu4yds5pgfyovo8ert

[root@cosmosia28 ~]# docker ps -a
CONTAINER ID   IMAGE              COMMAND                  CREATED          STATUS          PORTS     NAMES
627a83217258   archlinux:latest   "/bin/bash -c 'curl …"   3 minutes ago    Up 3 minutes              lb_terra2.1.fqrgb1job4blvan36ghbd5qy3
c6beab994e89   archlinux:latest   "/bin/bash -c 'curl …"   3 minutes ago    Up 3 minutes              lb_starname.1.sivbq964mcm7ghu3ghb51bk77
77ffd182defe   archlinux:latest   "/bin/bash -c 'curl …"   4 minutes ago    Up 4 minutes              lb_sentinel.1.u5teatmpsxicac13zxi3e8hs4
53719240df2f   archlinux:latest   "/bin/bash -c 'curl …"   5 minutes ago    Up 5 minutes              lb_quicksilver.1.cjws9rnuuy6gbjywlexp8y8vw
dd63a45f4920   archlinux:latest   "/bin/bash -c 'curl …"   6 minutes ago    Up 6 minutes              lb_passage.1.zlr50518wz1keah8ub6iq3vjf
5b31f9f0b57f   archlinux:latest   "/bin/bash -c 'curl …"   6 minutes ago    Up 6 minutes              lb_osmosis-testnet.1.q45c2pjgoa2077ln863sj5vxf
11652a742f69   archlinux:latest   "/bin/bash -c 'curl …"   7 minutes ago    Up 7 minutes              lb_likecoin.1.ax5zm3gry2i4x0a03nzpd8vkq
a6ce43fe109c   archlinux:latest   "/bin/bash -c 'curl …"   8 minutes ago    Up 8 minutes              lb_kujira.1.mtv0t07hx360jv024g6y94anm
9b392ae42778   archlinux:latest   "/bin/bash -c 'curl …"   9 minutes ago    Up 9 minutes              lb_juno.1.vsfnwzcgwdml0pbe8mwxohsuo
b51257aca075   archlinux:latest   "/bin/bash -c 'curl …"   9 minutes ago    Up 9 minutes              lb_irisnet.1.gfxx5xuvh38syhofrxmrbbc90
dd3b6df2b581   archlinux:latest   "/bin/bash -c 'curl …"   11 minutes ago   Up 11 minutes             lb_evmos-testnet-archive.1.tpf1sbk7763zy3ubhmej51e7w
b4561d82ca73   archlinux:latest   "/bin/bash -c 'curl …"   11 minutes ago   Up 11 minutes             lb_evmos-archive.1.ukam22jjig1lw6lmoq48biuco
b671d990b9e5   archlinux:latest   "/bin/bash -c 'curl …"   13 minutes ago   Up 13 minutes             lb_cryptoorgchain.1.vdpaure6vnnc8uollcaxs4xiw
4d022090d350   archlinux:latest   "/bin/bash -c 'curl …"   14 minutes ago   Up 14 minutes             lb_cheqd.1.ycvnws7dce1dmoguvhpizdka9
f7290352aae8   archlinux:latest   "/bin/bash -c 'curl …"   15 minutes ago   Up 15 minutes             lb_bandchain.1.1f8u5jmw6swikkn5zituuwpmc
b2b3005a0bfa   archlinux:latest   "/bin/bash -c 'curl …"   16 minutes ago   Up 16 minutes             lb_assetmantle.1.tw3uq3vvrprbqx8sbyv69i3g3

notional-labs / cosmosia

lb: dedicated nodes placement #255