Disaggregate model bundles

[nleiva]

Installing the cisco-ios-xr bundle currently takes hours. One thing that would improve the user experience is the option to selectively chose which models to include during the installation or provide all this in a pre-compiled binary.

You can also provide special bundles with let's say just BGP+ISIS+RPL models (and deal with dependencies). Users can potentially edit the existing bundles themselves, however I'm looking for alternatives to accelerate the user adoption and make all this transparent and as quick as possible for them.

[111pontes]

Please describe the environment under which install is taking hours. I just installed over a slow transoceanic VPN connection and a full install (all bundles, all dependencies) takes a bit over two minutes for me:

$ mkvirtualenv install-no-cache
# snipped
(install-no-cache) $ 
(install-no-cache) $ time pip --no-cache-dir install ydk-models-cisco-ios-xr
Collecting ydk-models-cisco-ios-xr
  Downloading ydk_models_cisco_ios_xr-6.2.1-py2.py3-none-any.whl (24.2MB)
    100% |████████████████████████████████| 24.2MB 673kB/s 
Collecting ydk-models-ietf>=0.1.1 (from ydk-models-cisco-ios-xr)
  Downloading ydk_models_ietf-0.1.1-py2.py3-none-any.whl (78kB)
    100% |████████████████████████████████| 81kB 912kB/s 
Collecting ncclient>=0.4.7 (from ydk-models-cisco-ios-xr)
  Downloading ncclient-0.5.3.tar.gz (63kB)
    100% |████████████████████████████████| 71kB 10.5MB/s 
Collecting ydk>=0.5.2 (from ydk-models-cisco-ios-xr)
  Downloading ydk-0.5.4-py2.py3-none-any.whl (76kB)
    100% |████████████████████████████████| 81kB 1.2MB/s 
Collecting lxml==3.4.4 (from ydk-models-cisco-ios-xr)
  Downloading lxml-3.4.4.tar.gz (3.5MB)
    100% |████████████████████████████████| 3.5MB 791kB/s 
Collecting enum34==1.1.3 (from ydk-models-cisco-ios-xr)
  Downloading enum34-1.1.3-py2.py3-none-any.whl (61kB)
    100% |████████████████████████████████| 61kB 4.2MB/s 
Collecting ydk-models-openconfig>=0.1.2 (from ydk-models-cisco-ios-xr)
  Downloading ydk_models_openconfig-0.1.2-py2.py3-none-any.whl (343kB)
    100% |████████████████████████████████| 348kB 689kB/s 
Requirement already satisfied: setuptools>0.6 in /home/saalvare/.virtualenvs/install-no-cache/lib/python2.7/site-packages (from ncclient>=0.4.7->ydk-models-cisco-ios-xr)
Collecting paramiko>=1.15.0 (from ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Downloading paramiko-2.1.2-py2.py3-none-any.whl (172kB)
    100% |████████████████████████████████| 174kB 688kB/s 
Requirement already satisfied: six in /home/saalvare/.virtualenvs/install-no-cache/lib/python2.7/site-packages (from ncclient>=0.4.7->ydk-models-cisco-ios-xr)
Collecting pybind11==2.0.1 (from ydk>=0.5.2->ydk-models-cisco-ios-xr)
  Downloading pybind11-2.0.1-py2.py3-none-any.whl (88kB)
    100% |████████████████████████████████| 92kB 8.5MB/s 
Requirement already satisfied: appdirs>=1.4.0 in /home/saalvare/.virtualenvs/install-no-cache/lib/python2.7/site-packages (from setuptools>0.6->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
Requirement already satisfied: packaging>=16.8 in /home/saalvare/.virtualenvs/install-no-cache/lib/python2.7/site-packages (from setuptools>0.6->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
Collecting pyasn1>=0.1.7 (from paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Downloading pyasn1-0.2.3-py2.py3-none-any.whl (53kB)
    100% |████████████████████████████████| 61kB 2.4MB/s 
Collecting cryptography>=1.1 (from paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Downloading cryptography-1.8.1.tar.gz (423kB)
    100% |████████████████████████████████| 430kB 826kB/s 
Requirement already satisfied: pyparsing in /home/saalvare/.virtualenvs/install-no-cache/lib/python2.7/site-packages (from packaging>=16.8->setuptools>0.6->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
Collecting idna>=2.1 (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Downloading idna-2.5-py2.py3-none-any.whl (55kB)
    100% |████████████████████████████████| 61kB 2.5MB/s 
Collecting asn1crypto>=0.21.0 (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Downloading asn1crypto-0.22.0-py2.py3-none-any.whl (97kB)
    100% |████████████████████████████████| 102kB 798kB/s 
Collecting ipaddress (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Downloading ipaddress-1.0.18-py2-none-any.whl
Collecting cffi>=1.4.1 (from cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Downloading cffi-1.10.0-cp27-cp27mu-manylinux1_x86_64.whl (392kB)
    100% |████████████████████████████████| 399kB 752kB/s 
Collecting pycparser (from cffi>=1.4.1->cryptography>=1.1->paramiko>=1.15.0->ncclient>=0.4.7->ydk-models-cisco-ios-xr)
  Downloading pycparser-2.17.tar.gz (231kB)
    100% |████████████████████████████████| 235kB 753kB/s 
Installing collected packages: pyasn1, idna, asn1crypto, enum34, ipaddress, pycparser, cffi, cryptography, paramiko, lxml, ncclient, pybind11, ydk, ydk-models-ietf, ydk-models-openconfig, ydk-models-cisco-ios-xr
  Running setup.py install for pycparser ... done
  Running setup.py install for cryptography ... done
  Running setup.py install for lxml ... done
  Running setup.py install for ncclient ... done
Successfully installed asn1crypto-0.22.0 cffi-1.10.0 cryptography-1.8.1 enum34-1.1.3 idna-2.5 ipaddress-1.0.18 lxml-3.4.4 ncclient-0.5.3 paramiko-2.1.2 pyasn1-0.2.3 pybind11-2.0.1 pycparser-2.17 ydk-0.5.4 ydk-models-cisco-ios-xr-6.2.1 ydk-models-ietf-0.1.1 ydk-models-openconfig-0.1.2

real    2m16.649s
user    1m17.036s
sys 0m4.668s
(install-no-cache) $
(install-no-cache) $ pip list | grep ydk
ydk                     0.5.4  
ydk-models-cisco-ios-xr 6.2.1  
ydk-models-ietf         0.1.1  
ydk-models-openconfig   0.1.2  
(install-no-cache) $ 

[nleiva]

Yeah, I should I've been a bit more specific. This is for CPP.

./generate.py --cpp --bundle profiles/bundles/cisco-ios-xr_6_1_2.json -> 20 min aprox. make this -> 5-6 hours (ymmv)

[111pontes]

Not sure I get the question. Is the issue generation time or installation time?

[ghost]

I believe Nicolas is referring to the compile time especially for cisco-ios-xr C++ bundle. Another solution is to provide pre-built binaries (tracked by #429).

[111pontes]

If the issue is compile time, that's correct. We'll be distributing installable packages.

[nleiva]

yep, that was it.. thanks @abhikeshav for translating my thoughts :-) ..Looking forward for the installable packages.