search

Onemind-Services-LLC / netbox-metatype-importer

Easy import Device and Module types from GitHub repo
Apache License 2.0
43 stars 7 forks source link

[Bug]: Connection reset by peer when clicking op "Load" on the Device Imports screen #25

Open litebito opened 1 year ago

litebito commented 1 year ago

NetBox plugin version

v0.3.0

NetBox version

Current: v3.7.1

But same issue for versions v3.7.0, v3.6.0, v3.5.3, ...

Steps to Reproduce

Centos Stream 8 The server has a system wide proxy set : In /etc/environment:

http_proxy=http://httppxxxx.srv.xxx.com:8080
https_proxy=http://httppxxxx.srv.xxx.com:8080
no_proxy=127.0.0.1,localhost

in Netbox Config:

HTTP_PROXIES = {
     'http': 'http://httppxxxx.srv.xxx.com:8080',
     'https': 'http://httppxxxx.srv.xxx.com:8080'
}

   'netbox_metatype_importer': {
            'github_token': '[mygithubtoken]'

In Netbox, with the plugin installed, clicking op "Load" on the Device Imports screen

Expected Behavior

the list of device types would load

Observed Behavior

Error message:

Server Error There was a problem with your request. Please contact an administrator.

The complete exception is provided below:

<class 'requests.exceptions.ConnectionError'>

('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Python version: 3.11.5 NetBox version: 3.7.0

abhi1693 commented 1 year ago

This does not look like an issue with the plugin unless you can provide detailed steps to reproduce it. Please recheck your deployment and make sure that the repo that you are trying to access is allowed via your proxy.

litebito commented 1 year ago

I can access the repo through the proxy (tested with wget)

abhi1693 commented 1 year ago

Could you give detailed information about your setup? Because I'm unable to reproduce this with a simple setup connecting to GitHub via a proxy.

litebito commented 1 year ago

This is the setup:

A virtual server CentOS Stream 8 Python v3.11, NetBox v3.5.7, Plugin v0.1.0 The server is located behind a firewall and a proxy. The proxy requires authentication (basic authentication seems to work)

I've set the global environment variables via /etc/environment:

proxy=http://user:password@proxy.company.com:8080
PROXY=http://user:password@proxy.company.com:8080
https_proxy=http://user:password@proxy.company.com:8080
http_proxy=http://user:password@proxy.company.com:8080
no_proxy=127.0.0.1,localhost
HTTPS_PROXY=http://user:password@proxy.company.com:8080

These settings work for wget, git, curl, dnf, the update script of netbox

Is this info helping?

litebito commented 7 months ago

Hi,

I updated the initial info in the topic start:

Testing:

curl https://raw.githubusercontent.com/netbox-community/devicetype-library/master/device-types/Cisco/2951-ISR.yaml
---
manufacturer: Cisco
model: 2951 ISR
part_number: CISCO2951/K9
slug: cisco-2951-isr
u_height: 2
is_full_depth: true
console-ports:
  - name: con 0
    type: rj-45
power-ports:
  - name: PS-1
    type: iec-60320-c14
    allocated_draw: 70
    maximum_draw: 340
interfaces:
  - name: GigabitEthernet0/0
    type: 1000base-t
  - name: GigabitEthernet0/1
    type: 1000base-t
  - name: GigabitEthernet0/2
    type: 1000base-t

So I can access github through the proxy

I also was able to turn on netbox logging, giving more insights:

`Internal Server Error: /plugins/netbox_metatype_importer/meta-device-types/load/
Traceback (most recent call last):
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 1096, in _validate_conn
    conn.connect()
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connection.py", line 782, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/ssl_.py", line 470, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/ssl_.py", line 514, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/ssl.py", line 1108, in _create
    self.do_handshake()
  File "/usr/lib64/python3.11/ssl.py", line 1379, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/netbox/venv/lib64/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connectionpool.py", line 1096, in _validate_conn
    conn.connect()
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/connection.py", line 782, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/ssl_.py", line 470, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/urllib3/util/ssl_.py", line 514, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/ssl.py", line 1108, in _create
    self.do_handshake()
  File "/usr/lib64/python3.11/ssl.py", line 1379, in do_handshake
    self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/netbox/venv/lib64/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/utilities/views.py", line 53, in dispatch
    return super().dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/netbox_metatype_importer/views.py", line 66, in post
    models = gh_api.get_tree()
             ^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/netbox_metatype_importer/gql.py", line 84, in get_tree
    data = self.get_query(query)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/netbox_metatype_importer/gql.py", line 66, in get_query
    response = self.session.post(url=self.url, json={'query': query})
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib64/python3.11/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Not Found: /favicon.ico
`

I can do actions like:

So for these actions, the proxy setup I have is working

abhi1693 commented 7 months ago

Can you check if you can access https://api.github.com/graphql using CURL? We are using GitHub APIs rather than raw objects

litebito commented 7 months ago

It looks like that is working:

# curl --request GET --url "https://api.github.com/graphql" --header "Authorization: Bearer [MY GITHUB TOKEN]" --header "X-GitHub-Api-Version: 2022-11-28" > curltest.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2469k  100 2469k    0     0  82870      0  0:00:30  0:00:30 --:--:--  413k

# cat curltest.txt | more
{"data":{"__schema":{"queryType":{"name":"Query"},"mutationType":{"name":"Mutation"},"subscriptionType":null,"types":[{"kind":"INPUT_OBJECT","name":"AbortQueuedMigra
tionsInput","description":"Autogenerated input type of AbortQueuedMigrations","fields":null,"inputFields":[{"name":"ownerId","description":"The ID of the organizatio
n that is running the migrations.","type":{"kind":"NON_NULL","name":null,"ofType":{"kind":"SCALAR","name":"ID","ofType":null}},"defaultValue":null},{"name":"clientMu
tationId","description":"A unique identifier for the client performing the mutation.","type":{"kind":"SCALAR","name":"String","ofType":null},"defaultValue":null}],"i
nterfaces":null,"enumValues":null,"possibleTypes":null},{"kind":"OBJECT","name":"AbortQueuedMigrationsPayload","description":"Autogenerated return type of AbortQueue
dMigrations","fields":[{"name":"clientMutationId","description":"A unique identifier for the client performing the mutation.","args":[],"type":{"kind":"SCALAR","name
":"String","ofType":null},"isDeprecated":false,"deprecationReason":null},

-------------- and so on, it's a long datastream
giffordjim commented 6 months ago

I have the same issue as well. Proxy in netbox works on all other connectivity except this plugin.

Internal Server Error: /plugins/netbox_metatype_importer/meta-module-types/load/

ConnectionError at /plugins/netbox_metatype_importer/meta-module-types/load/ HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa8b194bb50>: Failed to establish a new connection: [Errno 110] Connection timed out'))

curl --request GET --url "https://api.github.com/graphql" --header "Authorization: Bearer [MY GITHUB TOKEN]" --header "X-GitHub-Api-Version: 2022-11-28" > curltest.txt % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2483k 100 2483k 0 0 734k 0 0:00:03 0:00:03 --:--:-- 734k

litebito commented 6 months ago

we've tried with different proxies, with and without authentication, etc.... so far we've not found a way to resolve this issue

giffordjim commented 6 months ago

I was able to confirm, that the plugin when it tries to update doesn't honor the proxy settings. on my connection it was trying to go to 140.82.112.6 port 443 directly without using the proxy.

litebito commented 5 months ago

Hi, just checking of there is already a fix or solution coming up?