search

Chaffelson / nipyapi

A convenient Python wrapper for Apache NiFi
Other
246 stars 76 forks source link

Client Certificate Authorization Error for OIDC secured NIFI instance #197

Closed nimeshsubramanianv2 closed 1 year ago

nimeshsubramanianv2 commented 4 years ago

Description

We are trying to leverage nipyapi to automate flow deployment, parameter context, and user management. We have NIFI containers deployed in EC2 instances fronted by an ALB and a DNS in route 53. Due to security requirements NIFI has been secured through OIDC. We have been trying to authenticate to NIFI using client certs.

Authentication is working fine through the UI(OIDC). We recreated certs and added the user("CN=admin,OU=NIFI") with appropriate permissions through the initial admin user (controlled through OIDC).

What I Did

The error we get is: HTTP response body: Unknown user with identity 'anonymous'. Contact the system administrator

Nipyapi V0.14.1 gives us the same error.

Creation of certs:

../bin/tls-toolkit.sh standalone -n "nifi[1-3].nifi.internal" -K random -P random -S random -C CN=admin,OU=NIFI
password=$(cat CN=admin_OU=NIFI.password)
  echo $password
  openssl pkcs12 -in CN=admin_OU=NIFI.p12 -out admin_cert.der -nodes -password "pass:$password"
  openssl pkcs12 -in CN=admin_OU=NIFI.p12 -nodes -nocerts -out admin_key.key -password "pass:$password"
  perl -pe 'BEGIN{undef $/;} s|-----BEGIN PRIVATE KEY-----.*?-----END PRIVATE KEY-----|Removed private key|gs' admin_cert.der > admin_cert.pem

Code:

from __future__ import absolute_import
import logging
from pprint import pprint
from os import path
import nipyapi
from nipyapi.utils import DockerContainer

log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
logging.getLogger('nipyapi.utils').setLevel(logging.INFO)
logging.getLogger('nipyapi.security').setLevel(logging.INFO)
logging.getLogger('nipyapi.versioning').setLevel(logging.INFO)

# Uncomment the block below to enable debug logging
nipyapi.config.nifi_config.debug=True
nipyapi.config.registry_config.debug=True
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)

_basename = "nipyapi_secure"
_rc0 = _basename + '_reg_client_0'

d_network_name = 'securedemo'

secured_nifi_url = 'https://nifi.svcs.com/'
host_certs_path = '/Users/ns/workspace/nifi-cluster/keys'
nipyapi.nifi.configuration.verify_ssl = True
nipyapi.config.nifi_config.host = 'https://nifi.svcs.com/nifi-api'
nipyapi.security.set_service_ssl_context(
    service='nifi',
    ca_file=path.join(host_certs_path, 'nifi-cert.pem'),
    client_cert_file=path.join(host_certs_path, 'admin_cert.pem'),
    client_key_file=path.join(host_certs_path, 'admin_key.key'),
    client_key_password='ABCD'
)

nipyapi.utils.is_endpoint_up(secured_nifi_url)

temp = nipyapi.templates.list_all_templates(native=True)
pprint(temp)

Output: 

$ pipenv run python security.py
INFO:nipyapi.utils:Called is_endpoint_up with args {'endpoint_url': 'https://nifi.svcs.com/'}
2020-05-21 13:42:02,081 DEBUG Starting new HTTPS connection (1): nifi.svcs.com:443
2020-05-21 13:42:02,081 DEBUG Starting new HTTPS connection (1): nifi.svcs.com:443
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): nifi.svcs.com:443
send: b'GET / HTTP/1.1\r\nHost: nifi.svcs.com\r\nUser-Agent: python-requests/2.23.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Thu, 21 May 2020 17:42:02 GMT
header: Content-Type: text/html;charset=utf-8
header: Content-Length: 402
header: Connection: keep-alive
header: Set-Cookie: AWSALB=Z6LtmRZlmA/klYBHUKZHbO5HPBAQCu27EvvSq0rFqxPh0NI09bw4v5n9mdBM/t8fcvaZ836CmMzYreRi3K/hul4hLRwx1b8FxFSa41WhKuu2M76pTE5DGTcT7wDc; Expires=Thu, 28 May 2020 17:42:02 GMT; Path=/
header: Set-Cookie: AWSALB=Z6LtmRZlmA/klYBHUKZHbO5HPBAQCu27EvvSq0rFqxPh0NI09bw4v5n9mdBM/t8fcvaZ836CmMzYreRi3K/hul4hLRwx1b8FxFSa41WhKuu2M76pTE5DGTcT7wDc; Expires=Thu, 28 May 2020 17:42:02 GMT; Path=/
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Strict-Transport-Security: max-age=31540000
header: Vary: Accept-Encoding, User-Agent
header: Content-Encoding: gzip
header: Server: Jetty(9.4.19.v20190610)
2020-05-21 13:42:02,366 DEBUG https://nifi.svcs.com:443 "GET / HTTP/1.1" 200 402
2020-05-21 13:42:02,366 DEBUG https://nifi.svcs.com:443 "GET / HTTP/1.1" 200 402
DEBUG:urllib3.connectionpool:https://nifi.svcs.com:443 "GET / HTTP/1.1" 200 402
INFO:nipyapi.utils:Got 200 response from endpoint, returning True
2020-05-21 13:42:02,369 DEBUG Starting new HTTPS connection (1): nifi.svcs.com:443
2020-05-21 13:42:02,369 DEBUG Starting new HTTPS connection (1): nifi.svcs.com:443
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): nifi.svcs.com:443
send: b'GET /nifi-api/flow/templates HTTP/1.1\r\nHost: nifi.svcs.com\r\nAccept-Encoding: identity\r\nAccept: application/json\r\nContent-Type: application/json\r\nUser-Agent: Swagger-Codegen/1.0.0/python\r\n\r\n'
reply: 'HTTP/1.1 401 Unauthorized\r\n'
header: Date: Thu, 21 May 2020 17:42:02 GMT
header: Content-Type: text/plain
header: Content-Length: 73
header: Connection: keep-alive
header: Set-Cookie: AWSALB=2HL01Gbx0PPQIxYTO3YOClE9Db4MSTDUDSXGiBcj11LP2rTmfH1lt6gnqNBWTH3TZ+Bav77pg06Mis8JNJnxPn6tN4CFOK0NpxjqI51T/SaKn+vHIeCZS1JKNC8O; Expires=Thu, 28 May 2020 17:42:02 GMT; Path=/
header: Set-Cookie: AWSALB=2HL01Gbx0PPQIxYTO3YOClE9Db4MSTDUDSXGiBcj11LP2rTmfH1lt6gnqNBWTH3TZ+Bav77pg06Mis8JNJnxPn6tN4CFOK0NpxjqI51T/SaKn+vHIeCZS1JKNC8O; Expires=Thu, 28 May 2020 17:42:02 GMT; Path=/
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Strict-Transport-Security: max-age=31540000
header: Server: Jetty(9.4.19.v20190610)
header: X-ProxiedEntitiesAccepted: true
header: Vary: Accept-Encoding
2020-05-21 13:42:02,680 DEBUG https://nifi.svcs.com:443 "GET /nifi-api/flow/templates HTTP/1.1" 401 73
2020-05-21 13:42:02,680 DEBUG https://nifi.svcs.com:443 "GET /nifi-api/flow/templates HTTP/1.1" 401 73
DEBUG:urllib3.connectionpool:https://nifi.svcs.com:443 "GET /nifi-api/flow/templates HTTP/1.1" 401 73
DEBUG:nipyapi.nifi.rest:response body: Unknown user with identity 'anonymous'. Contact the system administrator.
Traceback (most recent call last):
  File "/Users/nifi.svcs.comns/.local/share/virtualenvs/nifi-cluster-Yw3ZJQrH/lib/python3.7/site-packages/nipyapi/utils.py", line 525, in rest_exceptions
    yield
  File "/Users/ns/.local/share/virtualenvs/nifi-cluster-Yw3ZJQrH/lib/python3.7/site-packages/nipyapi/templates.py", line 259, in list_all_templates
    templates = nipyapi.nifi.FlowApi().get_templates()
  File "/Users/ns/.local/share/virtualenvs/nifi-cluster-Yw3ZJQrH/lib/python3.7/site-packages/nipyapi/nifi/apis/flow_api.py", line 3801, in get_templates
    (data) = self.get_templates_with_http_info(**kwargs)
  File "/Users/ns/.local/share/virtualenvs/nifi-cluster-Yw3ZJQrH/lib/python3.7/site-packages/nipyapi/nifi/apis/flow_api.py", line 3875, in get_templates_with_http_info
    collection_formats=collection_formats)
  File "/Users/ns/.local/share/virtualenvs/nifi-cluster-Yw3ZJQrH/lib/python3.7/site-packages/nipyapi/nifi/api_client.py", line 326, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/Users/ns/.local/share/virtualenvs/nifi-cluster-Yw3ZJQrH/lib/python3.7/site-packages/nipyapi/nifi/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/Users/ns/.local/share/virtualenvs/nifi-cluster-Yw3ZJQrH/lib/python3.7/site-packages/nipyapi/nifi/api_client.py", line 349, in request
    headers=headers)
  File "/Users/ns/.local/share/virtualenvs/nifi-cluster-Yw3ZJQrH/lib/python3.7/site-packages/nipyapi/nifi/rest.py", line 233, in GET
    query_params=query_params)
  File "/Users/ns/.local/share/virtualenvs/nifi-cluster-Yw3ZJQrH/lib/python3.7/site-packages/nipyapi/nifi/rest.py", line 224, in request
    raise ApiException(http_resp=r)
nipyapi.nifi.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 21 May 2020 17:42:02 GMT', 'Content-Type': 'text/plain', 'Content-Length': '73', 'Connection': 'keep-alive', 'Set-Cookie': 'AWSALB=2HL01Gbx0PPQIxYTO3YOClE9Db4MSTDUDSXGiBcj11LP2rTmfH1lt6gnqNBWTH3TZ+Bav77pg06Mis8JNJnxPn6tN4CFOK0NpxjqI51T/SaKn+vHIeCZS1JKNC8O; Expires=Thu, 28 May 2020 17:42:02 GMT; Path=/, AWSALBCORS=2HL01Gbx0PPQIxYTO3YOClE9Db4MSTDUDSXGiBcj11LP2rTmfH1lt6gnqNBWTH3TZ+Bav77pg06Mis8JNJnxPn6tN4CFOK0NpxjqI51T/SaKn+vHIeCZS1JKNC8O; Expires=Thu, 28 May 2020 17:42:02 GMT; Path=/; SameSite=None; Secure', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Security-Policy': "frame-ancestors 'self'", 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=31540000', 'Server': 'Jetty(9.4.19.v20190610)', 'X-ProxiedEntitiesAccepted': 'true', 'Vary': 'Accept-Encoding'})
HTTP response body: Unknown user with identity 'anonymous'. Contact the system administrator.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "security.py", line 62, in <module>
    temp = nipyapi.templates.list_all_templates(native=True)
  File "/Users/ns/.local/share/virtualenvs/nifi-cluster-Yw3ZJQrH/lib/python3.7/site-packages/nipyapi/templates.py", line 259, in list_all_templates
    templates = nipyapi.nifi.FlowApi().get_templates()
  File "/Users/ns/.pyenv/versions/3.7.3/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/ns/.local/share/virtualenvs/nifi-cluster-Yw3ZJQrH/lib/python3.7/site-packages/nipyapi/utils.py", line 528, in rest_exceptions
    raise ValueError(e.body)
ValueError: Unknown user with identity 'anonymous'. Contact the system administrator.

Urgency

Please give a brief description of how critical this issue is to you.
We are trying to automate and harden our installation. Having this will help a lot of our use cases. Having read through other issues/PRs I know its doable. I just don't know what I am doing wrong. Help would be greatly appreciated and will help us move forward.

Chaffelson commented 4 years ago

Hi @nimeshsubramanianv2 sorry I've been out of the seat for a while dealing with Covid response work and childcare.

The anonymous response has come up before and I've been working to improve the security functions each time we spot a new case where it's not working as desired. Generally what is happening is your login and token are not being correctly set by the client, either through skipping a step or a bug on my part.

You can test if you are logged in using nipyapi.security.get_service_access_status, if my suspicion is correct you will see that you are not logged in.

Also if @alopresto has a minute to look over what you're doing here he may have suggestions.

Chaffelson commented 1 year ago

Closing as old, please reopen if the issue persists