Unable to connect to secure connection to Nifi in CDP Public Cloud

[LvffY]

• Nipyapi version: 0.19.0
• NiFi version: 1.16.0
• NiFi-Registry version: N/A
• Python version: 3.10.4
• Operating System: Windows

Description

I'm trying to request my nifi installation. My server is a Cloudera server behind a ranger/kerberos authentication.

What I Did

I'm trying to set up a simple request :

import nipyapi

nipyapi.config.nifi_config.host = "https://nifi_url/nifi-api"
nipyapi.config.nifi_config.username = "user"
nipyapi.config.nifi_config.password = "password"

parameters = nipyapi.parameters.get_parameter_context('my_id', identifier_type='id')
print(parameters)

With these lines I get the following errors

Traceback (most recent call last):
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\utils.py", line 640, in rest_exceptions
    yield
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\system.py", line 25, in get_system_diagnostics
    return nipyapi.nifi.SystemDiagnosticsApi().get_system_diagnostics()
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\nifi\apis\system_diagnostics_api.py", line 67, in get_system_diagnostics
    (data) = self.get_system_diagnostics_with_http_info(**kwargs)
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\nifi\apis\system_diagnostics_api.py", line 135, in get_system_diagnostics_with_http_info
    return self.api_client.call_api('/system-diagnostics', 'GET',
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\nifi\api_client.py", line 330, in call_api
    return self.__call_api(resource_path, method,
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\nifi\api_client.py", line 150, in __call_api
    response_data = self.request(method, url,
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\nifi\api_client.py", line 353, in request
    return self.rest_client.GET(url,
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\nifi\rest.py", line 240, in GET
    return self.request("GET", url,
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\nifi\rest.py", line 235, in request
    raise ApiException(http_resp=r)
nipyapi.nifi.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Server': 'nginx', 'Date': 'Tue, 26 Jul 2022 13:55:17 GMT', 'Content-Length': '0', 'Connection': 'keep-alive', 'WWW-Authenticate': 'BASIC realm="application"', 'x-response-nginx': 'true'})

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\plugins\modules\nifi\test_nifi.py", line 23, in <module>
    parameters = nipyapi.parameters.get_parameter_context('my_id', identifier_type='id')
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\utils.py", line 652, in wrapper
    return f(*args, **kwargs)
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\parameters.py", line 54, in get_parameter_context
    enforce_min_ver('1.10.0')
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\utils.py", line 572, in enforce_min_ver
    if check_version(min_version, service=service) == 1:
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\utils.py", line 525, in check_version
    nipyapi.system.get_nifi_version_info().ni_fi_version
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\system.py", line 59, in get_nifi_version_info
    diags = get_system_diagnostics()
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\system.py", line 24, in get_system_diagnostics
    with nipyapi.utils.rest_exceptions():
  File "C:\Python310\lib\contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\nipyapi\utils.py", line 643, in rest_exceptions
    _raise(ValueError(e.body), e)
  File "C:\Users\aberges\IdeaProjects\GEODIS\ansible_collections\itcorp_datalayer\deploy_app\venv\lib\site-packages\future\utils\__init__.py", line 403, in raise_from
    exec(execstr, myglobals, mylocals)
  File "<string>", line 1, in <module>
ValueError

Process finished with exit code 1

On the other side, if I'm doing simple rest api calls, it works :

import requests
from pprint import pprint

auth = ("user", "pwd")

req = requests.get(f"https://nifi_url/nifi-api/parameter-contexts/my_id", auth=auth)

 pprint(req.json()["component"]["parameters"])

I think that others requests are made but I'm not aware of which requests.

Urgency

It is not really blocking but I would really like to use this API instead of the RAW REST API. The latest I have an answer on this, the toughest it will be on my side to replace REST API calls by the API.

[Chaffelson]

Two initial thoughts:

1. I think you are being blocked by NiPy attempting to fetch get_system_diagnostics - probably the user you are using doesn't have that authz in Ranger. That call is attempting to determine the version of NiFi to handle some compatibility switches between NiFi and NiFi-Registry, we should probably have it fail more gracefully when the permission is not available.
2. We don't test on Python 3.10 yet as I've seen some issues with urllib3 that I haven't had time to unpack. If you continue to see issues I also suggest trying python >3.6,<=3.9 to be sure

[LvffY]

@Chaffelson Thanks for your answer :)

• I don't think it's related to python version. I tried with python 3.9.13 and it didn't work either, with the same issue.
• I tried to request /system-diagnostics and it succeeds in REST API while failed with the Python API

import requests
from pprint import pprint

auth = ("user", "pwd")

req = requests.get(f"https://nifi_url/system-diagnostics", auth=auth)

pprint(req.json())

I looked in my ranger audit, but didn't see any failure :(

Now I'm a bit confused ^^'

[Chaffelson]

Ah ok, can I suggest you try using nipyapi.utils.set_endpoint

Nipy doesn't expect you to modify the underlying configs directly, although you can in many cases. For simple activities like you are trying, I created helper methods to ensure that login and logout are completed correctly and with the right security context.

[LvffY]

@Chaffelson I did not see this function, should be useful !

Unfortunately, I still got the same error :(

import nipyapi

nipyapi.utils.set_endpoint(endpoint_url="https://nifi_url/nifi-api", username="user", password="password")

parameters = nipyapi.parameters.get_parameter_context('my_id', identifier_type='id')
print(parameters)

[Chaffelson]

Try:

import nipyapi

nipyapi.utils.set_endpoint(endpoint_url="https://nifi_url/nifi-api", username="user", password="password", login=True)

parameters = nipyapi.parameters.get_parameter_context('my_id', identifier_type='id')
print(parameters)

[LvffY]

@Chaffelson Same error :(

[Chaffelson]

Ok I think we need to sync up on what is not working, hit me up on the Apache NiFi slack and we can diagnose it

[LvffY]

After a call with @Chaffelson (thanks again ;)) we have a working setup :

We think that our problem is only related to our environment (which is a CDP Public Cloud from Cloudera) but in case anyone fall into this, you need to setup the force_basic_auth to True

To work, I did the following code :

import nipyapi

############ For Nifi server
nipyapi.config.nifi_config.force_basic_auth = True
nipyapi.config.nifi_config.username = "user"
nipyapi.config.nifi_config.password = "password"
nipyapi.utils.set_endpoint(endpoint_url=nifi_api)
print(nipyapi.canvas.get_root_pg_id())

########### For Nifi registry
nipyapi.config.registry_config.username = "user"
nipyapi.config.registry_config.password = "password"
nipyapi.config.registry_config.force_basic_auth = True
nipyapi.utils.set_endpoint(self.nifi_reg_endpoint)

[Chaffelson]

I think we should add an enhancement to the set_endpoint method to allow a flag for basic_auth usage