search

Chaffelson / nipyapi

A convenient Python wrapper for Apache NiFi
Other
245 stars 76 forks source link

can't connect to nif by https #222

Closed error0x1 closed 12 months ago

error0x1 commented 4 years ago

Description

We have a few nodes with nifi. Our nifi work by https. I tryed connect to nifi by nipyapi via https, but get error CERTIFICATE_VERIFY_FAILED.

What I Did

import nipyapi nipyapi.config.nifi_config.host = 'https://host12.sg.sf.ru:9443/nifi-api' nipyapi.security.service_login(service='nifi', username='OU-MAX-SN', password='XZ4232', bool_response=True, auth_type='eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJPVVQtTHVzaGluLU1WIiwiaXNzIjoiTGRhcFByb3ZpZGVyIiwiYXVkIjoiTGRhcFByb3ZpZGVyIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiT1VULUx1c2hpbi1NViIsImtpZCI6MSwiZXhwIjoxNjAxNjEwNjA3LCJpYXQiOjE2MDE1Njc0MDd9.e78OSiBdDgBvdiz0XiqFZ76bWCkEwX2FfVv7LrKsxXA') True

nipyapi.canvas.get_root_pg_id() 2020-10-01 18:55:10,540 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status 2020-10-01 18:55:10,540 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status 2020-10-01 18:55:10,556 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status 2020-10-01 18:55:10,556 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status 2020-10-01 18:55:10,573 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status 2020-10-01 18:55:10,573 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 978, in _validate_conn conn.connect() File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connection.py", line 371, in connect sslcontext=context, File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/util/ssl.py", line 384, in ssl_wrap_socket return context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 773, in init self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1033, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 645, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/canvas.py", line 41, in get_root_pg_id return nipyapi.nifi.FlowApi().get_process_group_status('root') \ File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/apis/flow_api.py", line 2723, in get_process_group_status (data) = self.get_process_group_status_with_http_info(id, kwargs) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/apis/flow_api.py", line 2813, in get_process_group_status_with_http_info collection_formats=collection_formats) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/api_client.py", line 326, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/api_client.py", line 153, in __call_api _request_timeout=_request_timeout) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/api_client.py", line 349, in request headers=headers) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/rest.py", line 233, in GET query_params=query_params) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/rest.py", line 207, in request headers=headers) File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/request.py", line 76, in request method, url, fields=fields, headers=headers, urlopen_kw File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/request.py", line 97, in request_encode_url return self.urlopen(method, url, extra_kw) File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/poolmanager.py", line 336, in urlopen response = conn.urlopen(method, u.request_uri, kw) File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 767, in urlopen response_kw File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 767, in urlopen response_kw File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 767, in urlopen **response_kw File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 727, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/util/retry.py", line 439, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='host12.sg.sf.ru', port=9443): Max retries exceeded with url: /nifi-api/flow/process-groups/root/status (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))

Urgency

it's blocking our Production environment.

Chaffelson commented 4 years ago

Did you try disabling SSL verification in config.py?

On Tue, 13 Oct 2020, 11:55 error0x1, notifications@github.com wrote:

  • Nipyapi version: last
  • NiFi version: 1.9.0
  • NiFi-Registry version: -
  • Python version: 2.7/3.6
  • Operating System: Read Hat

Description

We have a few nodes with nifi. Our nifi work by https. I tryed connect to nifi by nipyapi via https, but get error CERTIFICATE_VERIFY_FAILED.

What I Did

import nipyapi nipyapi.config.nifi_config.host = 'https://host12.sg.sf.ru:9443/nifi-api' nipyapi.security.service_login(service='nifi', username='OU-MAX-SN', password='XZ4232', bool_response=True, auth_type='eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJPVVQtTHVzaGluLU1WIiwiaXNzIjoiTGRhcFByb3ZpZGVyIiwiYXVkIjoiTGRhcFByb3ZpZGVyIiwicHJlZmVycmVkX3VzZXJuYW1lIjoiT1VULUx1c2hpbi1NViIsImtpZCI6MSwiZXhwIjoxNjAxNjEwNjA3LCJpYXQiOjE2MDE1Njc0MDd9.e78OSiBdDgBvdiz0XiqFZ76bWCkEwX2FfVv7LrKsxXA') True

nipyapi.canvas.get_root_pg_id() 2020-10-01 18:55:10,540 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status 2020-10-01 18:55:10,540 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status 2020-10-01 18:55:10,556 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status 2020-10-01 18:55:10,556 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status WARNING:urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status 2020-10-01 18:55:10,573 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status 2020-10-01 18:55:10,573 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)': /nifi-api/flow/process-groups/root/status Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked, File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 978, in

validate_conn conn.connect() File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connection.py", line 371, in connect ssl_context=context, File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/util/ssl.py", line 384, in ssl_wrap_socket return context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket _context=self, _session=session) File "/usr/lib64/python3.6/ssl.py", line 773, in init self.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 1033, in do_handshake self._sslobj.do_handshake() File "/usr/lib64/python3.6/ssl.py", line 645, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/canvas.py", line 41, in get_root_pg_id return nipyapi.nifi.FlowApi().get_process_group_status('root') File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/apis/flow_api.py", line 2723, in get_process_group_status (data) = self.get_process_group_status_with_http_info(id, kwargs) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/apis/flow_api.py", line 2813, in get_process_group_status_with_http_info collection_formats=collection_formats) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/api_client.py", line 326, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/api_client.py", line 153, in __call_api _request_timeout=_request_timeout) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/api_client.py", line 349, in request headers=headers) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/rest.py", line 233, in GET query_params=query_params) File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/rest.py", line 207, in request headers=headers) File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/request.py", line 76, in request method, url, fields=fields, headers=headers, urlopen_kw File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/request.py", line 97, in request_encode_url return self.urlopen(method, url, extra_kw) File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/poolmanager.py", line 336, in urlopen response = conn.urlopen(method, u.request_uri, kw) File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 767, in urlopen response_kw File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 767, in urlopen response_kw File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 767, in urlopen **response_kw File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/connectionpool.py", line 727, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/usr/local/lib/python3.6/site-packages/urllib3-1.25.10-py3.6.egg/urllib3/util/retry.py", line 439, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=' host12.sg.sf.ru', port=9443): Max retries exceeded with url: /nifi-api/flow/process-groups/root/status (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),)) Urgency

it's blocking our Production environment.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Chaffelson/nipyapi/issues/222, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZAZOEYAMRQH736D47IF7DSKQWZTANCNFSM4SOTPGJA .

error0x1 commented 4 years ago

Did you try disabling SSL verification in config.py?

Thanks for infotmation. I changed that parametr to false , but my next try was finishet with fail.

nipyapi.security.service_login(service='nifi', username='OU-MAX-SN', password='XZ4232', bool_response=True, auth_type='basic')

True

nipyapi.canvas.get_root_pg_id()

Traceback (most recent call last):

File "", line 1, in

File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/canvas.py", line 41, in get_root_pg_id

return nipyapi.nifi.FlowApi().get_process_group_status('root') \

File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/apis/flow_api.py", line 2723, in get_process_group_status

(data) = self.get_process_group_status_with_http_info(id, **kwargs)

File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/apis/flow_api.py", line 2813, in get_process_group_status_with_http_info

collection_formats=collection_formats)

File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/api_client.py", line 326, in call_api

_return_http_data_only, collection_formats, _preload_content, _request_timeout)

File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/api_client.py", line 153, in __call_api

_request_timeout=_request_timeout)

File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/api_client.py", line 349, in request

headers=headers)

File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/rest.py", line 233, in GET

query_params=query_params)

File "/usr/local/lib/python3.6/site-packages/nipyapi-0.14.3-py3.6.egg/nipyapi/nifi/rest.py", line 224, in request

raise ApiException(http_resp=r)

nipyapi.nifi.rest.ApiException: (401)

Reason: Unauthorized

HTTP response headers: HTTPHeaderDict({'Date': 'Tue, 13 Oct 2020 15:55:36 GMT', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Security-Policy': "frame-ancestors 'self'", 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=31540000', 'Content-Type': 'text/plain', 'Vary': 'Accept-Encoding', 'Content-Length': '73', 'Server': 'Jetty(9.4.11.v20180605)'})

HTTP response body: Unknown user with identity 'anonymous'. Contact the system administrator.

Chaffelson commented 4 years ago

Right, I think you're hitting a bug I've recently been nailing down with authentication and authorization (always fun to debug). From speaking to other members of the NiFi community I have learned that NiFi will use presented authentication methods in the order it wants, so if you provide say a certificate and a username / password, it will use the certificate. This means I'm going to have to refactor how the security in NiPy works to be more consistent.

For this situation, what is the exact auth method you want to use?

error0x1 commented 4 years ago

What do you mean? We use LDAP for auth in nifi like a user and certs fot auth like a node

error0x1 commented 4 years ago

Right, I think you're hitting a bug I've recently been nailing down with authentication and authorization (always fun to debug). From speaking to other members of the NiFi community I have learned that NiFi will use presented authentication methods in the order it wants, so if you provide say a certificate and a username / password, it will use the certificate. This means I'm going to have to refactor how the security in NiPy works to be more consistent.

For this situation, what is the exact auth method you want to use? Sorry, can you help me? We have 3 nodes nifi. We have auth by ldap via AD and thay work on httpS. And whan I tryed connect by nipyapi with disablet ssl verification I have error nipyapi.security.service_login(service='nifi', username='OU-MAX-SN', password='XZ4232', bool_response=True, auth_type='basic')

I have error like a https://github.com/Chaffelson/nipyapi/issues/222#issuecomment-707845809

Can you help me?

Chaffelson commented 12 months ago

Closing as old, please reopen if the issue persists