netbox-community / Device-Type-Library-Import

This library is intended to assist with importing device and module types into NetBox from the NetBox Community DeviceType-Library
https://github.com/netbox-community/devicetype-library
MIT License
248 stars 75 forks source link

HTTP Client closing connection seemingly randomly #63

Closed ndom91 closed 1 year ago

ndom91 commented 1 year ago

So in general the importer seems to be working, it gets through the first few devices, but then without error always dies with the following error:

Error: urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Open for stack trace / full output ```python Traceback (most recent call last): File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen chunked=chunked, File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.7/http/client.py", line 1336, in getresponse response.begin() File "/usr/lib/python3.7/http/client.py", line 306, in begin version, status, reason = self._read_status() File "/usr/lib/python3.7/http/client.py", line 275, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/urllib3/connectionpool.py", line 756, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/urllib3/util/retry.py", line 532, in increment raise six.reraise(type(error), error, _stacktrace) File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/urllib3/packages/six.py", line 769, in reraise raise value.with_traceback(tb) File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/urllib3/connectionpool.py", line 706, in urlopen chunked=chunked, File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/urllib3/connectionpool.py", line 445, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse() File "/usr/lib/python3.7/http/client.py", line 1336, in getresponse response.begin() File "/usr/lib/python3.7/http/client.py", line 306, in begin version, status, reason = self._read_status() File "/usr/lib/python3.7/http/client.py", line 275, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./nb-dt-import.py", line 853, in main() File "./nb-dt-import.py", line 826, in main createDeviceTypes(deviceTypes, nb) File "./nb-dt-import.py", line 707, in createDeviceTypes dt.id, nb) File "./nb-dt-import.py", line 554, in create_module_bays devicetype_id=device_type File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/pynetbox/core/endpoint.py", line 225, in filter ret = [self._response_loader(i) for i in req.get()] File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/pynetbox/core/query.py", line 300, in get return req_all() File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/pynetbox/core/query.py", line 279, in req_all req = self._make_call(add_params=add_params) File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/pynetbox/core/query.py", line 247, in _make_call params=params, json=data File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/requests/sessions.py", line 555, in get return self.request('GET', url, **kwargs) File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/opt/netbox-pi/Netbox-Device-Type-Library-Import/lib/python3.7/site-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) ```

I'm watching the netbox container http logs during the time and see nothing out of the ordinary, just all of the GET and POST requests coming and going, all 200 responses.

Last few http requests from netbox logs before error (open for more details) ```logtalk [pid: 443|app: 0|req: 142/279] 172.19.0.1 () {30 vars in 460 bytes} [Sat Dec 31 16:20:56 2022] GET /api/dcim/manufacturers/ => generated 15843 bytes in 253 msecs (HTTP/1.1 200) 9 headers in 329 bytes (1 switches on core 0) [pid: 443|app: 0|req: 143/280] 172.19.0.1 () {30 vars in 497 bytes} [Sat Dec 31 16:20:56 2022] GET /api/dcim/manufacturers/?limit=92&offset=50 => generated 13442 bytes in 214 msecs (HTTP/1.1 200) 9 headers in 329 bytes (1 switches on core 0) [pid: 443|app: 0|req: 144/281] 172.19.0.1 () {30 vars in 458 bytes} [Sat Dec 31 16:20:56 2022] GET /api/dcim/device-types/ => generated 4271 bytes in 152 msecs (HTTP/1.1 200) 9 headers in 328 bytes (1 switches on core 0) [pid: 443|app: 0|req: 145/282] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:20:56 2022] GET /api/dcim/module-bay-templates/?devicetype_id=1 => generated 4524 bytes in 99 msecs (HTTP/1.1 200) 9 headers in 328 bytes (1 switches on core 0) [pid: 443|app: 0|req: 146/283] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:20:57 2022] GET /api/dcim/module-bay-templates/?devicetype_id=2 => generated 3045 bytes in 90 msecs (HTTP/1.1 200) 9 headers in 328 bytes (1 switches on core 0) [pid: 443|app: 0|req: 147/284] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:20:57 2022] GET /api/dcim/module-bay-templates/?devicetype_id=3 => generated 1548 bytes in 87 msecs (HTTP/1.1 200) 9 headers in 328 bytes (1 switches on core 0) [pid: 443|app: 0|req: 148/285] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:20:57 2022] GET /api/dcim/module-bay-templates/?devicetype_id=4 => generated 6040 bytes in 107 msecs (HTTP/1.1 200) 9 headers in 328 bytes (1 switches on core 0) [pid: 443|app: 0|req: 149/286] 172.19.0.1 () {30 vars in 503 bytes} [Sat Dec 31 16:20:57 2022] GET /api/dcim/interface-templates/?devicetype_id=5 => generated 15481 bytes in 150 msecs (HTTP/1.1 200) 9 headers in 329 bytes (1 switches on core 0) [pid: 443|app: 0|req: 150/287] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:20:57 2022] GET /api/dcim/power-port-templates/?devicetype_id=5 => generated 649 bytes in 101 msecs (HTTP/1.1 200) 9 headers in 327 bytes (1 switches on core 0) [pid: 443|app: 0|req: 151/288] 172.19.0.1 () {30 vars in 509 bytes} [Sat Dec 31 16:20:57 2022] GET /api/dcim/console-port-templates/?devicetype_id=5 => generated 1197 bytes in 130 msecs (HTTP/1.1 200) 9 headers in 328 bytes (1 switches on core 0) [pid: 443|app: 0|req: 152/289] 172.19.0.1 () {30 vars in 503 bytes} [Sat Dec 31 16:20:57 2022] GET /api/dcim/interface-templates/?devicetype_id=6 => generated 52 bytes in 72 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 153/290] 172.19.0.1 () {34 vars in 512 bytes} [Sat Dec 31 16:20:57 2022] POST /api/dcim/interface-templates/ => generated 609 bytes in 91 msecs (HTTP/1.1 201) 10 headers in 350 bytes (1 switches on core 0) [pid: 443|app: 0|req: 154/291] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:20:57 2022] GET /api/dcim/power-port-templates/?devicetype_id=6 => generated 52 bytes in 68 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 155/292] 172.19.0.1 () {34 vars in 515 bytes} [Sat Dec 31 16:20:58 2022] POST /api/dcim/power-port-templates/ => generated 1205 bytes in 119 msecs (HTTP/1.1 201) 10 headers in 351 bytes (1 switches on core 0) [pid: 443|app: 0|req: 156/293] 172.19.0.1 () {30 vars in 509 bytes} [Sat Dec 31 16:20:58 2022] GET /api/dcim/console-port-templates/?devicetype_id=6 => generated 52 bytes in 61 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 157/294] 172.19.0.1 () {34 vars in 518 bytes} [Sat Dec 31 16:20:58 2022] POST /api/dcim/console-port-templates/ => generated 545 bytes in 78 msecs (HTTP/1.1 201) 10 headers in 350 bytes (1 switches on core 0) [pid: 443|app: 0|req: 158/295] 172.19.0.1 () {30 vars in 509 bytes} [Sat Dec 31 16:20:58 2022] GET /api/dcim/power-outlet-templates/?devicetype_id=6 => generated 52 bytes in 71 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 159/296] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:20:58 2022] GET /api/dcim/power-port-templates/?devicetype_id=6 => generated 1255 bytes in 108 msecs (HTTP/1.1 200) 9 headers in 328 bytes (1 switches on core 0) [pid: 443|app: 0|req: 160/297] 172.19.0.1 () {34 vars in 519 bytes} [Sat Dec 31 16:20:58 2022] POST /api/dcim/power-outlet-templates/ => generated 5443 bytes in 371 msecs (HTTP/1.1 201) 10 headers in 351 bytes (1 switches on core 0) [pid: 443|app: 0|req: 161/298] 172.19.0.1 () {34 vars in 500 bytes} [Sat Dec 31 16:20:58 2022] POST /api/dcim/device-types/ => generated 608 bytes in 145 msecs (HTTP/1.1 201) 11 headers in 408 bytes (1 switches on core 0) [pid: 443|app: 0|req: 162/299] 172.19.0.1 () {30 vars in 503 bytes} [Sat Dec 31 16:20:59 2022] GET /api/dcim/interface-templates/?devicetype_id=7 => generated 52 bytes in 65 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 163/300] 172.19.0.1 () {34 vars in 512 bytes} [Sat Dec 31 16:20:59 2022] POST /api/dcim/interface-templates/ => generated 606 bytes in 94 msecs (HTTP/1.1 201) 10 headers in 350 bytes (1 switches on core 0) [pid: 443|app: 0|req: 164/301] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:20:59 2022] GET /api/dcim/power-port-templates/?devicetype_id=7 => generated 52 bytes in 70 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 165/302] 172.19.0.1 () {34 vars in 514 bytes} [Sat Dec 31 16:20:59 2022] POST /api/dcim/power-port-templates/ => generated 601 bytes in 81 msecs (HTTP/1.1 201) 10 headers in 350 bytes (1 switches on core 0) [pid: 443|app: 0|req: 166/303] 172.19.0.1 () {30 vars in 509 bytes} [Sat Dec 31 16:20:59 2022] GET /api/dcim/console-port-templates/?devicetype_id=7 => generated 52 bytes in 61 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 167/304] 172.19.0.1 () {34 vars in 518 bytes} [Sat Dec 31 16:20:59 2022] POST /api/dcim/console-port-templates/ => generated 535 bytes in 85 msecs (HTTP/1.1 201) 10 headers in 350 bytes (1 switches on core 0) [pid: 443|app: 0|req: 168/305] 172.19.0.1 () {30 vars in 509 bytes} [Sat Dec 31 16:20:59 2022] GET /api/dcim/power-outlet-templates/?devicetype_id=7 => generated 52 bytes in 64 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 169/306] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:20:59 2022] GET /api/dcim/power-port-templates/?devicetype_id=7 => generated 651 bytes in 104 msecs (HTTP/1.1 200) 9 headers in 327 bytes (1 switches on core 0) [pid: 443|app: 0|req: 170/307] 172.19.0.1 () {34 vars in 520 bytes} [Sat Dec 31 16:20:59 2022] POST /api/dcim/power-outlet-templates/ => generated 17479 bytes in 1237 msecs (HTTP/1.1 201) 10 headers in 352 bytes (1 switches on core 0) [pid: 443|app: 0|req: 171/308] 172.19.0.1 () {34 vars in 499 bytes} [Sat Dec 31 16:21:01 2022] POST /api/dcim/device-types/ => generated 561 bytes in 138 msecs (HTTP/1.1 201) 11 headers in 408 bytes (1 switches on core 0) [pid: 443|app: 0|req: 172/309] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:21:01 2022] GET /api/dcim/power-port-templates/?devicetype_id=8 => generated 52 bytes in 72 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 173/310] 172.19.0.1 () {34 vars in 514 bytes} [Sat Dec 31 16:21:01 2022] POST /api/dcim/power-port-templates/ => generated 600 bytes in 78 msecs (HTTP/1.1 201) 10 headers in 350 bytes (1 switches on core 0) [pid: 443|app: 0|req: 174/311] 172.19.0.1 () {30 vars in 509 bytes} [Sat Dec 31 16:21:01 2022] GET /api/dcim/power-outlet-templates/?devicetype_id=8 => generated 52 bytes in 68 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 175/312] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:21:01 2022] GET /api/dcim/power-port-templates/?devicetype_id=8 => generated 650 bytes in 100 msecs (HTTP/1.1 200) 9 headers in 327 bytes (1 switches on core 0) [pid: 443|app: 0|req: 176/313] 172.19.0.1 () {34 vars in 519 bytes} [Sat Dec 31 16:21:01 2022] POST /api/dcim/power-outlet-templates/ => generated 4713 bytes in 325 msecs (HTTP/1.1 201) 10 headers in 351 bytes (1 switches on core 0) [pid: 443|app: 0|req: 177/314] 172.19.0.1 () {34 vars in 500 bytes} [Sat Dec 31 16:21:01 2022] POST /api/dcim/device-types/ => generated 629 bytes in 137 msecs (HTTP/1.1 201) 11 headers in 408 bytes (1 switches on core 0) [pid: 443|app: 0|req: 178/315] 172.19.0.1 () {30 vars in 503 bytes} [Sat Dec 31 16:21:01 2022] GET /api/dcim/interface-templates/?devicetype_id=9 => generated 52 bytes in 73 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 179/316] 172.19.0.1 () {34 vars in 512 bytes} [Sat Dec 31 16:21:02 2022] POST /api/dcim/interface-templates/ => generated 615 bytes in 83 msecs (HTTP/1.1 201) 10 headers in 350 bytes (1 switches on core 0) [pid: 443|app: 0|req: 180/317] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:21:02 2022] GET /api/dcim/power-port-templates/?devicetype_id=9 => generated 52 bytes in 70 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 181/318] 172.19.0.1 () {34 vars in 514 bytes} [Sat Dec 31 16:21:02 2022] POST /api/dcim/power-port-templates/ => generated 610 bytes in 78 msecs (HTTP/1.1 201) 10 headers in 350 bytes (1 switches on core 0) [pid: 443|app: 0|req: 182/319] 172.19.0.1 () {30 vars in 509 bytes} [Sat Dec 31 16:21:02 2022] GET /api/dcim/console-port-templates/?devicetype_id=9 => generated 52 bytes in 62 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 183/320] 172.19.0.1 () {34 vars in 518 bytes} [Sat Dec 31 16:21:02 2022] POST /api/dcim/console-port-templates/ => generated 544 bytes in 77 msecs (HTTP/1.1 201) 10 headers in 350 bytes (1 switches on core 0) [pid: 443|app: 0|req: 184/321] 172.19.0.1 () {30 vars in 509 bytes} [Sat Dec 31 16:21:02 2022] GET /api/dcim/power-outlet-templates/?devicetype_id=9 => generated 52 bytes in 66 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 185/322] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:21:02 2022] GET /api/dcim/power-port-templates/?devicetype_id=9 => generated 660 bytes in 103 msecs (HTTP/1.1 200) 9 headers in 327 bytes (1 switches on core 0) [pid: 443|app: 0|req: 186/323] 172.19.0.1 () {34 vars in 520 bytes} [Sat Dec 31 16:21:02 2022] POST /api/dcim/power-outlet-templates/ => generated 17695 bytes in 1285 msecs (HTTP/1.1 201) 10 headers in 352 bytes (1 switches on core 0) [pid: 443|app: 0|req: 187/324] 172.19.0.1 () {34 vars in 500 bytes} [Sat Dec 31 16:21:03 2022] POST /api/dcim/device-types/ => generated 567 bytes in 142 msecs (HTTP/1.1 201) 11 headers in 409 bytes (1 switches on core 0) [pid: 443|app: 0|req: 188/325] 172.19.0.1 () {30 vars in 505 bytes} [Sat Dec 31 16:21:04 2022] GET /api/dcim/interface-templates/?devicetype_id=10 => generated 52 bytes in 69 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 189/326] 172.19.0.1 () {34 vars in 512 bytes} [Sat Dec 31 16:21:04 2022] POST /api/dcim/interface-templates/ => generated 611 bytes in 87 msecs (HTTP/1.1 201) 10 headers in 350 bytes (1 switches on core 0) [pid: 443|app: 0|req: 190/327] 172.19.0.1 () {30 vars in 507 bytes} [Sat Dec 31 16:21:04 2022] GET /api/dcim/power-port-templates/?devicetype_id=10 => generated 52 bytes in 76 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) [pid: 443|app: 0|req: 191/328] 172.19.0.1 () {34 vars in 515 bytes} [Sat Dec 31 16:21:04 2022] POST /api/dcim/power-port-templates/ => generated 1209 bytes in 120 msecs (HTTP/1.1 201) 10 headers in 351 bytes (1 switches on core 0) [pid: 443|app: 0|req: 192/329] 172.19.0.1 () {30 vars in 511 bytes} [Sat Dec 31 16:21:04 2022] GET /api/dcim/console-port-templates/?devicetype_id=10 => generated 52 bytes in 65 msecs (HTTP/1.1 200) 9 headers in 326 bytes (1 switches on core 0) ```

Any idea what could be causing this?

Netbox: 3.4.1 Netbox-Device-Type-Library-Import: main latest (31.12.22)

ndom91 commented 1 year ago

It keeps getting further and further along in the process every time I run it, so it doesn't seem to be failing on any particular manufacturer / device :thinking:

ndom91 commented 1 year ago

Bumping up the timeout and retries in ..lib/python3.7/site-packages/requests/adapters.py:448 seems to have helped it get further along, but still fails eventually before making it all the way through every time :/

ndom91 commented 1 year ago

Silly workaround - a bash script to retry importing until the exit code is 0, i.e. success :joy:

#!/bin/bash

echo "Entering python venv"
source "$(pwd)/bin/activate"

echo "Starting loop.."
until python3 nb-dt-import.py; [[ $? -eq 0 ]];
do
  echo "*** RETRYING IMPORT ***"
  sleep 1
done

echo "Netbox Device-Type Import successful"
danner26 commented 1 year ago

@ndom91 is this still an issue?

ndom91 commented 1 year ago

I believe so, but I was able to get around it and complete my latest import with the workaround I mentioned above.

I'll close this issue for now then. If anyone else stumbles upon it they can reopen or just create a new one 👍

mcaulifn commented 1 year ago

Seems like it might be the same issue?

~/shared/device_sync$ ./nb-dt-import.py
Package devicetype-library is already installed, updating /srv/netbox/shared/device_sync/repo
104 Vendors Found
2306 Device-Types Found
Traceback (most recent call last):
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connection.py", line 454, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/urllib3/connection.py", line 454, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/netbox/shared/device_sync/./nb-dt-import.py", line 53, in <module>
    main()
  File "/srv/netbox/shared/device_sync/./nb-dt-import.py", line 25, in main
    netbox.create_device_types(device_types)
  File "/srv/netbox/shared/device_sync/netbox_api.py", line 97, in create_device_types
    self.device_types.create_interfaces(device_type["interfaces"], dt.id)
  File "/srv/netbox/shared/device_sync/netbox_api.py", line 196, in create_interfaces
    existing_interfaces = {str(item): item for item in self.netbox.dcim.interface_templates.filter(
  File "/srv/netbox/shared/device_sync/netbox_api.py", line 196, in <dictcomp>
    existing_interfaces = {str(item): item for item in self.netbox.dcim.interface_templates.filter(
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/pynetbox/core/response.py", line 127, in __next__
    next(self.response), self.endpoint.api, self.endpoint
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/pynetbox/core/query.py", line 280, in get
    req = self._make_call(add_params=add_params)
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/pynetbox/core/query.py", line 230, in _make_call
    req = getattr(self.http_session, verb)(
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/srv/netbox/shared/device_sync/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
danner26 commented 1 year ago

It appears that the connection is being reset by your netbox server. Do you have a rate limiter or something weird happening on the NetBox server? I suggesting checking the server logs, as well as any proxy/web filtering/firewall logs.

danner26 commented 1 year ago

I have not experienced this on any of the test machines I have utilized either. What OS are you running, along with what version of python?

mcaulifn commented 1 year ago

This is directly on the instance, no proxy/LB/etc.

The request logs show the following when it works:

[pid: 2757|app: 0|req: 109/109] 127.0.0.1 () {30 vars in 467 bytes} [Fri Jun  2 17:45:43 2023] GET /api/dcim/manufacturers/?limit=0 => generated 360 bytes in 84 msecs (HTTP/1.1 200) 9 headers in 340 bytes (1 switches on core 0)

And the following when it doesn't:

[pid: 2757|app: 0|req: 104/104] 127.0.0.1 () {30 vars in 365 bytes} [Fri Jun  2 17:44:44 2023] GET /api/ => generated 455 bytes in 33 msecs (HTTP/1.1 200) 10 headers in 332 bytes (1 switches on core 0)

If I manually replicate using pynetbox, the API calls work.

$ python
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pynetbox
>>> nb = pynetbox.api("http://localhost", "xxxx")
>>> {str(item): item for item in nb.dcim.manufacturers.all()}
{'blah': blah}

Trying to use the NetBox class gets me the above:

>>> from netbox_api import NetBox
>>> import settings
Package devicetype-library is already installed, updating /srv/netbox/shared/device_sync/repo
>>> nb2 = NetBox(settings)
Traceback (most recent call last):
...

Replicating just the get_manufacturers function gets me the actual data:

>>> def get_manufacturers():
...     return {str(item): item for item in nb.dcim.manufacturers.all()}
...
>>> get_manufacturers()
{'blah': blah}

I can open a new issue if not related.

sjwk commented 1 year ago

I'm getting the exact same error on my test netbox instance. Netbox is running as a docker instance (using lscr.io/linuxserver/netbox:latest) on Ubuntu 20.04 host. The device type library import scripts running natively on the host in a venv.

It imports some devices then errors out. Watching the logs from the netbox container, I can see the API request counter increasing - the most recent time it managed 350 before crashing out, the time before that only 11 requests.

Aaron-von-Awesome commented 11 months ago

I am receiving this error as well. I am using the docker container to perform the import. I have modified what @ndom91 provided in order to get items to import. Sometimes it fails quickly, other times it manages to import a large number of records:

Workaround Script: Edit: I ended up having to loop over each vendor one at a time; even then there were constant errors. However, after enough retries, I managed to get all the device types imported.

#!/bin/bash

serverOrIp=$1
port=$2
token=$3
vendorList="3com,a10,adva,apc,avm,actiontec,adtran,albis-elcon,alcatel-lucent,\
  allied telesis,alpha,apple,arista,avocent,brocade,calix,canon,certa,check point,\
  ciena,cisco,citrix,colable,corning,cryptopro,cyberpower,d-link,datadirect networks,\
  datacom,datto,deciso,dell,delta,digi,digital loggers,eaton,edgecore,eltex,engenius,\
  ericsson,extreme networks,f5,fs,factor-ts,firebrick,fortinet,fujitsu,generic,hpe,huawei,\
  ibm,infinera,infoblox,infotecs,inspur,intel,juniper,lancom,lenovo,leviton,mellanox,\
  mikrotik,mimosa networks,nec,neousys,netapp,netgate,netgear,netonix,nokia,nvidia,opengear,\
  pc engines,palo alto,panduit,parks,peplink,pluribus,qct,qnap,qtech,qotom,rad,raritan,\
  raspberry pi,riello,riverbed,rockwell automation,rohde & schwarz,ruckus,snr,solid optics,\
  sonicwall,sophos,supermicro,synology,tp-link,teltonika,testing,thales,trendnet,ubiquiti,\
  v-solution,vertiv,wti,yamaha,zpe,zte,zyxel,ghipsystem"

IFS=',' read -ra vendorsArray <<< "$vendorList"

echo "Starting outer loop.."

for vendor in "${vendorsArray[@]}"; do
  echo "Starting inner loop for vendor: $vendor"
  echo "docker run --rm -e \"NETBOX_URL=http://${serverOrIp}:${port}/\" -e \"NETBOX_TOKEN=${token}\" -e \"VENDORS=${vendor}\" ghcr.io/minitriga/netbox-device-type-library-import"

  until docker run --rm -e "NETBOX_URL=http://${serverOrIp}:${port}/" -e "NETBOX_TOKEN=${token}" -e "VENDORS=${vendor}" ghcr.io/minitriga/netbox-device-type-library-import; [[ $? -eq 0 ]];
  do
    echo "*** RETRYING IMPORT ***"
    sleep 3s
  done

  echo "Netbox Device-Type Import successful for vendor: $vendor"
done

echo "All vendors processed"

Error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/local/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.9/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.9/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/local/lib/python3.9/http/client.py", line 320, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.9/http/client.py", line 289, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/nb-dt-import.py", line 853, in <module>
    main()
  File "/app/nb-dt-import.py", line 826, in main
    createDeviceTypes(deviceTypes, nb)
  File "/app/nb-dt-import.py", line 694, in createDeviceTypes
    createConsoleServerPorts(
  File "/app/nb-dt-import.py", line 348, in createConsoleServerPorts
    all_consoleserverports = {str(item): item for item in nb.dcim.console_server_port_templates.filter(devicetype_id=deviceType)}
  File "/usr/local/lib/python3.9/site-packages/pynetbox/core/endpoint.py", line 225, in filter
    ret = [self._response_loader(i) for i in req.get()]
  File "/usr/local/lib/python3.9/site-packages/pynetbox/core/query.py", line 300, in get
    return req_all()
  File "/usr/local/lib/python3.9/site-packages/pynetbox/core/query.py", line 279, in req_all
    req = self._make_call(add_params=add_params)
  File "/usr/local/lib/python3.9/site-packages/pynetbox/core/query.py", line 245, in _make_call
    req = getattr(self.http_session, verb)(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
danner26 commented 11 months ago

I wonder if it is the docker bit that is messing with it. We are still working on updating those docker images since they havent been rebuilt since the last maintainer

Jaz108 commented 3 months ago

I wonder if it is the docker bit that is messing with it. We are still working on updating those docker images since they havent been rebuilt since the last maintainer

No I am facing the same issue while using standalone python script on CentOS 7 VM

sa6kad commented 1 week ago

I am still having this issue. Running script on WSL and netbox on docker on a VM on the same machine. This is just a test setup, but it seems like there is some kind of rate limit on the Netbox that drops the connection.

sa6kad commented 1 week ago

Silly workaround - a bash script to retry importing until the exit code is 0, i.e. success 😂

#!/bin/bash

echo "Entering python venv"
source "$(pwd)/bin/activate"

echo "Starting loop.."
until python3 nb-dt-import.py; [[ $? -eq 0 ]];
do
  echo "*** RETRYING IMPORT ***"
  sleep 1
done

echo "Netbox Device-Type Import successful"

This works until there is some big number added, after that the script fails while going through existing deviecs, i.e. not reaching those that are not added. A workaround is to randomize the list of device types that need to be added :). Altogether this is a silly solution, but it works:) So, int the netbox_api.py file add import random and then randomize 'device_types_to_add' within the create_device_types method, like this:

import random
def create_device_types(self, device_types_to_add):
    random.shuffle(device_types_to_add)                    # <--- Just add this line 
    for device_type in device_types_to_add:
    .................