search

CentaurusInfra / mizar

Mizar – Experimental, High Scale and High Performance Cloud Network https://mizar.readthedocs.io
https://mizar.readthedocs.io
GNU General Public License v2.0
111 stars 50 forks source link

Mizar yaml takes excessive time to deploy on first time deployment on fresh VM #556

Closed vinaykul closed 3 years ago

vinaykul commented 3 years ago

Mizar yaml deployment on very first time on fresh VM takes excessive amount of time to come up and running.

It seems to be stuck in the Init container. It is building something according to top output.

Repro steps:

  1. Install two fresh Ubuntu 18.04 VMs in AWS, and clone the mizar repo and run bootstrap script on both.
  2. Use kubeadm to create a two node cluster.
  3. Deploy mizar bia etc/deploy/deploy.mizar.yaml
  4. Observe the time taken for Mizar pods to come to running state.

Run top to see the processes that are running.

vinaykul commented 3 years ago

The reason for this slow initial startup is:

k logs mizar-daemon-sp9lt -c node-init
...
...
  Downloading https://files.pythonhosted.org/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl (77kB)
Building wheels for collected packages: grpcio, grpcio-tools, luigi, pyroute2, scapy, tornado, pyroute2.core, pyroute2.ethtool, pyroute2.ipdb, pyroute2.ipset, pyroute2.ndb, pyroute2.nftables, pyroute2.nslink
  Running setup.py bdist_wheel for grpcio: started
  Running setup.py bdist_wheel for grpcio: still running...
  Running setup.py bdist_wheel for grpcio: still running...
  Running setup.py bdist_wheel for grpcio: still running...
  Running setup.py bdist_wheel for grpcio: still running...
  Running setup.py bdist_wheel for grpcio: still running...
  Running setup.py bdist_wheel for grpcio: still running...
  Running setup.py bdist_wheel for grpcio: still running...
vinaykul commented 3 years ago

This might help: https://github.com/grpc/grpc/issues/24026