search

networktocode / network-importer

The network importer is a tool/library to analyze and/or synchronize an existing network with a Network Source of Truth (SOT), it's designed to be idempotent and by default it's only showing the difference between the running network and the remote SOT.
Other
169 stars 41 forks source link

Missing `packaging` module import causing exception #224

Closed dteycheney closed 3 years ago

dteycheney commented 3 years ago

Environment

Package                      Version
---------------------------- -------------
aiofiles                     0.6.0
aiohttp                      3.7.2
aiohttp-swagger              1.0.15
async-lru                    1.0.2
async-timeout                3.0.1
attrs                        21.2.0
bcrypt                       3.2.0
bidict                       0.21.2
certifi                      2021.5.30
cffi                         1.14.6
chardet                      3.0.4
charset-normalizer           2.0.4
ciscoconfparse               1.5.46
click                        7.1.2
colorama                     0.4.4
commonmark                   0.9.1
ConfigArgParse               0.15.2
cryptography                 3.3.1
deepdiff                     5.5.0
Deprecated                   1.2.12
diffsync                     1.2.0
dill                         0.3.4
distro                       1.6.0
dnspython                    2.1.0
future                       0.18.2
genie                        21.4
genie.libs.clean             21.4
genie.libs.conf              21.4
genie.libs.filetransferutils 21.4
genie.libs.health            21.4.2
genie.libs.ops               21.4
genie.libs.parser            21.4
genie.libs.sdk               21.4.2
idna                         3.2
ipaddress                    1.0.23
Jinja2                       2.11.3
jsonpickle                   2.0.0
junit-xml                    1.9
junos-eznc                   2.6.2
loguru                       0.5.3
lxml                         4.6.3
MarkupSafe                   1.1.1
multidict                    5.1.0
mypy-extensions              0.4.3
napalm                       3.3.1
ncclient                     0.6.9
netaddr                      0.8.0
netconan                     0.12.2
netmiko                      3.4.0
network-importer             3.0.0
nornir                       3.1.1
nornir-napalm                0.1.2
nornir-netmiko               0.1.1
nornir-utils                 0.1.2
ntc-templates                2.2.0
numpy                        1.21.2
ordered-set                  4.0.2
pandas                       1.1.5
paramiko                     2.7.2
passlib                      1.7.4
pathspec                     0.9.0
pip                          21.2.4
prettytable                  2.1.0
psutil                       5.8.0
pyats                        21.4
pyats.aereport               21.4
pyats.aetest                 21.4
pyats.async                  21.4
pyats.connections            21.4
pyats.datastructures         21.4
pyats.easypy                 21.4
pyats.kleenex                21.4
pyats.log                    21.4
pyats.reporter               21.4
pyats.results                21.4
pyats.tcl                    21.4
pyats.topology               21.4
pyats.utils                  21.4
pybatfish                    2021.2.26.830
pycparser                    2.20
pydantic                     1.8.2
pyeapi                       0.8.4
pyftpdlib                    1.5.6
Pygments                     2.10.0
PyNaCl                       1.4.0
pynautobot                   1.0.3
pynetbox                     5.3.1
pyparsing                    2.4.7
pyserial                     3.5
python-dateutil              2.8.2
python-engineio              3.13.2
python-socketio              4.6.0
pytz                         2021.1
PyYAML                       5.4.1
requests                     2.26.0
requests-toolbelt            0.9.1
rich                         9.13.0
ruamel.yaml                  0.16.13
ruamel.yaml.clib             0.2.6
scp                          0.13.6
setuptools                   49.2.1
simplejson                   3.17.3
six                          1.16.0
structlog                    20.2.0
tenacity                     8.0.1
termcolor                    1.1.0
textfsm                      1.1.2
tftpy                        0.8.2
toml                         0.10.2
tqdm                         4.62.1
transitions                  0.8.8
typing-extensions            3.10.0.0
unicon                       21.4
unicon.plugins               21.4.2
urllib3                      1.26.6
wcwidth                      0.2.5
wheel                        0.37.0
wrapt                        1.12.1
xmltodict                    0.12.0
yamllint                     1.26.2
yamlordereddictloader        0.4.0
yarl                         1.6.3

Steps to Reproduce

  1. Setup a toml file. In my example, I have a file called yy.toml: 
    
[main]
import_ips = true
import_prefixes = true
import_cabling = "config"
import_vlans = "config"

backend = "nautobot"

[logs] level = "debug"

[inventory.settings] address = "https://10.X.Y.Z"

2. Execute the following command to check connectivity:

network-importer check --update-configs --config yy.toml

Below is the output I get, with the traceback:

2021-08-16 14:26:53,837 - network-importer - INFO - Updating configuration from devices .. 2021-08-16 14:26:53,845 - network-importer - DEBUG - Executing dispatcher for ANONYMISED-DEV-L27-C44-R1 (iosxe) 2021-08-16 14:26:53,846 - network-importer - DEBUG - Found driver network_importer.drivers.default 2021-08-16 14:26:53,846 - network-importer - DEBUG - Executing dispatcher for ANONYMISED-DEV-L27-C95-S1 (iosxe) 2021-08-16 14:26:53,846 - network-importer - DEBUG - Executing dispatcher for ANONYMISED-DEV-L27-N93-S2 (nxos) 2021-08-16 14:26:53,847 - network-importer - DEBUG - Found driver network_importer.drivers.default 2021-08-16 14:26:53,847 - network-importer - DEBUG - Executing dispatcher for ANONYMISED-DEV-L30-C93-S1 (iosxe) 2021-08-16 14:26:53,847 - network-importer - DEBUG - Found driver network_importer.drivers.default 2021-08-16 14:26:53,848 - network-importer - DEBUG - Executing dispatcher for ANONYMISED-DEV-L30-C93-S2 (iosxe) 2021-08-16 14:26:53,848 - network-importer - DEBUG - Found driver network_importer.drivers.default 2021-08-16 14:26:53,848 - network-importer - DEBUG - Found driver network_importer.drivers.default 2021-08-16 14:26:53,857 - network-importer - DEBUG - Executing get_config for ANONYMISED-DEV-L27-C95-S1 (iosxe) 2021-08-16 14:26:53,857 - network-importer - DEBUG - Executing get_config for ANONYMISED-DEV-L27-C44-R1 (iosxe) 2021-08-16 14:26:53,857 - network-importer - DEBUG - Executing get_config for ANONYMISED-DEV-L30-C93-S1 (iosxe) 2021-08-16 14:26:53,858 - network-importer - DEBUG - Executing get_config for ANONYMISED-DEV-L30-C93-S2 (iosxe) 2021-08-16 14:26:53,858 - network-importer - DEBUG - Executing get_config for ANONYMISED-DEV-L27-N93-S2 (nxos) 2021-08-16 14:26:59,719 - network-importer - INFO - ANONYMISED-DEV-L27-C44-R1 | Latest config file already present ... 2021-08-16 14:27:01,066 - network-importer - INFO - ANONYMISED-DEV-L30-C93-S1 | Latest config file already present ... 2021-08-16 14:27:02,152 - network-importer - INFO - ANONYMISED-DEV-L30-C93-S2 | Latest config file already present ... 2021-08-16 14:27:02,549 - network-importer - INFO - ANONYMISED-DEV-L27-N93-S2 | Latest config file already present ... 2021-08-16 14:27:02,988 - network-importer - INFO - ANONYMISED-DEV-L27-C95-S1 | Latest config file already present ... 2021-08-16 14:27:02,989 - network-importer - INFO - Import SOT Model Traceback (most recent call last): File "/home/xdteycheney/.pyenv/versions/net_nautobot_toolkit/bin/network-importer", line 8, in sys.exit(main()) File "/home/xdteycheney/.pyenv/versions/3.8.6/envs/net_nautobot_toolkit/lib/python3.8/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/home/xdteycheney/.pyenv/versions/3.8.6/envs/net_nautobot_toolkit/lib/python3.8/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/xdteycheney/.pyenv/versions/3.8.6/envs/net_nautobot_toolkit/lib/python3.8/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/xdteycheney/.pyenv/versions/3.8.6/envs/net_nautobot_toolkit/lib/python3.8/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/xdteycheney/.pyenv/versions/3.8.6/envs/net_nautobot_toolkit/lib/python3.8/site-packages/click/core.py", line 610, in invoke return callback(args, *kwargs) File "/home/localuser/.pyenv/versions/3.8.6/envs/net_nautobot_toolkit/lib/python3.8/site-packages/network_importer/cli.py", line 148, in check ni.init(limit=limit) File "/home/localuser/.pyenv/versions/3.8.6/envs/net_nautobot_toolkit/lib/python3.8/site-packages/network_importer/performance.py", line 67, in timed result = method(args, **kw) File "/home/localuser/.pyenv/versions/3.8.6/envs/net_nautobot_toolkit/lib/python3.8/site-packages/network_importer/main.py", line 106, in init sot_adapter = getattr(importlib.import_module(".".join(sot_path[0:-1])), sot_path[-1]) File "/home/localuser/.pyenv/versions/3.8.6/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 783, in exec_module File "", line 219, in _call_with_frames_removed File "/home/localuser/.pyenv/versions/3.8.6/envs/net_nautobot_toolkit/lib/python3.8/site-packages/network_importer/adapters/nautobot_api/adapter.py", line 6, in from packaging.version import Version, InvalidVersion ModuleNotFoundError: No module named 'packaging'


The code is expecting a `packaging` module, or perhaps there is a local import/file missing?
https://github.com/networktocode/network-importer/blob/v3.0.0/network_importer/adapters/netbox_api/adapter.py#L7
3.

<!-- What did you expect to happen? -->
### Expected Behavior
I expected the command-line tool to check the connectivity

<!-- What happened instead? -->
### Observed Behavior
I got the exception above.
dteycheney commented 3 years ago

Further analysis

After thinking that packaging was a local import, I found that it's actually a pypi module.

I then installed the packaging module:

pip3 install packaging

It looks like this commit replaced semver with packaging but the poetry file wasn't updated with packaging

I then ran the tool again and it's working.

Suspected solution

Update the relevant files with the packaging details which were altered in the commit:

https://github.com/networktocode/network-importer/commit/72b5715cbb605e408282120b9ad2504cd6470df3

dgarros commented 3 years ago

Based on this discussion it looks like it could be related to your version of pip and setuptools https://gist.github.com/hangtwenty/b9820fe204eebb0cc5b9aba49f3c8b22

We need to work on re-enabling the integration tests in CI to properly validate that

dteycheney commented 3 years ago

Thanks @dgarros, just wanted to confirm that you are going to re-enable integration tests for these?

dgarros commented 3 years ago

@dteycheney Yes we need to re-enable the integration tests, but it might take some time because we want to migrate to Github Actions as well. I'll try to reenable travis CI in the meantime but not sure if it will work, we had some issues in the past.

dteycheney commented 3 years ago

All good @dgarros , happy with the workaround for now

dgarros commented 3 years ago

I'm gonna close this one, @jvanderaa reenabled all CI tests on GHA in #226