Closed zpe-jose-deleon closed 1 year ago
zpe-reneneumann
commented
1 year ago @zpe-jose-deleon Can you provide your phyton code? Further to that, what Nodegrid version are you using? The error was caused by a Webserver Timeout. @zpe-diegor is there a better way of catching the Webserver timeouts and failing more gracefully?
zpe-jose-deleon
commented
1 year ago @zpe-diegor I have attached the code I used. Run command: python3 ztp-nodegrid.py --target nodegrid_ngbsr-102 < configs/ngbsr-102.yml
Replace target with your salt target. If you are not root, prefix with sudo
zpe-diegor
commented
1 year ago @zpe-jose-deleon Thanks for reporting this issue. Although I could not execute your example:
yocto@builder-ng5 ~/workdir/ztp-nodegrids $ sudo python3 ztp-nodegrid.py --target nodegrid < configs/ngbsr-102.cli
[ERROR ] An un-handled exception was caught by Salt's global exception handler:
TypeError: 'type' object is not subscriptable
Traceback (most recent call last):
File "ztp-nodegrid.py", line 15, in <module>
ng.load(data)
File "/home/yocto/workdir/ztp-nodegrids/nodegrid_utils/__init__.py", line 33, in load
self.from_settings(data)
File "/home/yocto/workdir/ztp-nodegrids/nodegrid_utils/__init__.py", line 36, in from_settings
def _recurse(d: dict, chain: list[str], value: str):
TypeError: 'type' object is not subscriptableDepending on Nodegrid versions, the CLI has a bad performance and timeouts so this can be expected in some cases, for instance Managed Devices, but this is not the case here.
From traceback, this issue happened after we sent control-d, paths are being processed but we expected a CLI prompt within the timeout of 30s only, but import processing took more time than that and timed out at at BACKPLANE0:
before (last 100 chars): b'ctions/hotspot Result: succeeded\r\nPath: /settings/network_connections/BACKPLANE0 Result: succeededAs talked with @zpe-leonardof we will apply a fix to calculate a timeout number for the number of different paths included of the import blob to avoid this issue.
zpe-diegor
commented
1 year ago To replicate, one can make a import_settings using template provided, at ztp-nodegrids.tar.gz on ztp-nodegrids/configs/ngbsr-102.cli
tar -xf ztp-nodegrids.tar.gz
cd ztp-nodegrids
cp configs/ngbsr-102.cli /srv/salt/import_test_large.cli
sudo salt <TARGET> nodegrid.import_settings_file salt://import_test_large.cli --timeout 1500
# current results: traceback and did not finished import procedure
# expected results: True or False (some import lines may be rejected)
I attempted same import directly with nodegrid CLI and it succeeds, however it does reject two paths, otherwise it succeeds everywhere else. But when run through the salt-proxy, there is traceback
{'nodegrid_ngbsr-102': 'The minion function caused an exception: Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/salt/metaproxy/proxy.py", line 482, in thread_return\n return_data = minion_instance.executors[fname](\n File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in call\n return self.loader.run(run_func, *args, kwargs)\n File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run\n return self._last_context.run(self._run_as, _func_or_method, *args, *kwargs)\n File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as\n return _func_or_method(args, kwargs)\n File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 10, in execute\n return func(*args, kwargs)\n File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in call\n return self.loader.run(run_func, *args, *kwargs)\n File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run\n return self._last_context.run(self._run_as, _func_or_method, args, kwargs)\n File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as\n return _func_or_method(args, kwargs)\n File "/var/cache/salt/proxy/extmods/nodegrid_ngbsr-102/modules/nodegrid.py", line 80, in import_settings\n return proxy["nodegrid.import_settings"](command, kwargs) # pylint: disable=undefined-variable\n File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in call\n return self.loader.run(run_func, args, kwargs)\n File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run\n return self._last_context.run(self._run_as, _func_or_method, *args, *kwargs)\n File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as\n return _func_or_method(args, kwargs)\n File "/var/cache/salt/proxy/extmods/nodegrid_ngbsr-102/proxy/nodegrid.py", line 748, in import_settings\n output_dict = _exec_import_settings(conn_obj, import_settings_list, True)\n File "/var/cache/salt/proxy/extmods/nodegrid_ngbsr-102/proxy/nodegrid.py", line 371, in _exec_import_settings\n conn_obj.expect_exact(\'/]# \')\n File "/usr/lib/python3/dist-packages/pexpect/spawnbase.py", line 421, in expect_exact\n return exp.expect_loop(timeout)\n File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 181, in expect_loop\n return self.timeout(e)\n File "/usr/lib/python3/dist-packages/pexpect/expect.py", line 144, in timeout\n raise exc\npexpect.exceptions.TIMEOUT: Timeout exceeded.\n<pexpect.pty_spawn.spawn object at 0x7f3b4ad92c20>\ncommand: /usr/bin/ssh\nargs: [\'/usr/bin/ssh\', \'-o\', \'UserKnownHostsFile=/dev/null\', \'-o\', \'StrictHostKeyChecking=no\', \'-o\', \'LogLevel=QUIET\', \'admin@192.168.74.239\']\nbuffer (last 100 chars): b\'ed\r\n\'\nbefore (last 100 chars): b\'ctions/hotspot Result: succeeded\r\nPath: /settings/network_connections/BACKPLANE0 Result: succeeded\r\n\'\nafter: <class \'pexpect.exceptions.TIMEOUT\'>\nmatch: None\nmatch_index: None\nexitstatus: None\nflag_eof: False\npid: 1772\nchild_fd: 25\nclosed: False\ntimeout: 60\ndelimiter: <class \'pexpect.exceptions.EOF\'>\nlogfile: None\nlogfile_read: <_io.BytesIO object at 0x7f3b5a9f7600>\nlogfile_send: None\nmaxread: 2000\nignorecase: False\nsearchwindowsize: None\ndelaybeforesend: 0.05\ndelayafterclose: 0.1\ndelayafterterminate: 0.1\nsearcher: searcher_string:\n 0: b\'/]# \'\n'}