Import Flow Version Failing With Anonymous Identity

Nipyapi version: 0.16.1
NiFi version: 1.11.4
NiFi-Registry version: 0.8.0
Python version: 3.7.4
Operating System: Windows

Description

The import_flow_version command is returning 401 errors despite security.service_login returning true. Of note I also have the global_ssl_verify = false set in config.py since the certs arent available for us on local. Other commands such as get_registry_bucket() work fine

What I Did

import nipyapi nipyapi.config.registry_config.host = nipyapi.security.service_login(service='registry', username=user, password=pass, bool_response=True, auth_type='token') True nipyapi.versioning.import_flow_version("5d7bc617-b702-4ba2-8168-f7bf78d3c391", encoded_flow=None, file_path='./tmp/flow.json', flow_name="name", flow_id=None)

Traceback (most recent call last): File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\utils.py", line 604, in rest_exceptions yield File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\versioning.py", line 470, in create_flow version_count=0 File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\apis\bucket_flows_api.py", line 67, in create_flow (data) = self.create_flow_with_http_info(bucket_id, body, **kwargs) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\apis\bucket_flows_api.py", line 154, in create_flow_with_http_info collection_formats=collection_formats) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\api_client.py", line 326, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\api_client.py", line 153, in __call_api _request_timeout=_request_timeout) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\api_client.py", line 371, in request body=body) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\rest.py", line 268, in POST body=body) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\rest.py", line 224, in request raise ApiException(http_resp=r) nipyapi.registry.rest.ApiException: (401) Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 28 Jan 2021 19:30:27 GMT', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'Strict-Transport-Security': 'max-age=31540000 ; includeSubDomains', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Security-Policy': "frame-ancestors 'self'", 'Content-Length': '73', 'Server': 'Jetty(9.4.19.v20190610)'}) HTTP response body: Unknown user with identity 'anonymous'. Contact the system administrator.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "", line 1, in File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\versioning.py", line 670, in import_flow_version flow_name=flow_name File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\versioning.py", line 470, in create_flow version_count=0 File "C:\Program Files\Python37\lib\contextlib.py", line 130, in exit self.gen.throw(type, value, traceback) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\utils.py", line 607, in rest_exceptions _raise(ValueError(e.body), e) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\future\utils__init__.py", line 403, in raise_from exec(execstr, myglobals, mylocals) File "", line 1, in ValueError: Unknown user with identity 'anonymous'. Contact the system administrator.

Urgency

Blocking work on our nifi flow migration

Can you try the current main branch? I merged a fix for a similar issue but it's not released yet.

On Thu, 28 Jan 2021, 19:39 crhicko, notifications@github.com wrote:

Nipyapi version: 0.16.1

NiFi version: 1.11.4

NiFi-Registry version: 0.8.0

Python version: 3.7.4

Operating System: Windows

Description

The import_flow_version command is returning 401 errors despite security.service_login returning true. Of note I also have the global_ssl_verify = false set in config.py since the certs arent available for us on local. Other commands such as get_registry_bucket() work fine What I Did

import nipyapi nipyapi.config.registry_config.host = nipyapi.security.service_login(service='registry', username=user, password=pass, bool_response=True, auth_type='token') True nipyapi.versioning.import_flow_version("5d7bc617-b702-4ba2-8168-f7bf78d3c391", encoded_flow=None, file_path='./tmp/flow.json', flow_name="name", flow_id=None)

Traceback (most recent call last): File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\utils.py", line 604, in rest_exceptions yield File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\versioning.py", line 470, in create_flow version_count=0 File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\apis\bucket_flows_api.py", line 67, in create_flow (data) = self.create_flow_with_http_info(bucket_id, body, **kwargs) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\apis\bucket_flows_api.py", line 154, in create_flow_with_http_info collection_formats=collection_formats) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\api_client.py", line 326, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\api_client.py", line 153, in __call_api _request_timeout=_request_timeout) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\api_client.py", line 371, in request body=body) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\rest.py", line 268, in POST body=body) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\rest.py", line 224, in request raise ApiException(http_resp=r) nipyapi.registry.rest.ApiException: (401) Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 28 Jan 2021 19:30:27 GMT', 'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'Strict-Transport-Security': 'max-age=31540000 ; includeSubDomains', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Security-Policy': "frame-ancestors 'self'", 'Content-Length': '73', 'Server': 'Jetty(9.4.19.v20190610)'}) HTTP response body: Unknown user with identity 'anonymous'. Contact the system administrator.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "", line 1, in File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\versioning.py", line 670, in import_flow_version flow_name=flow_name File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\versioning.py", line 470, in create_flow version_count=0 File "C:\Program Files\Python37\lib\contextlib.py", line 130, in exit self.gen.throw(type, value, traceback) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\utils.py", line 607, in rest_exceptions

raise(ValueError(e.body), e) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\future\utils_init.py", line 403, in raise_from exec(execstr, myglobals, mylocals) File "", line 1, in ValueError: Unknown user with identity 'anonymous'. Contact the system administrator. Urgency

Blocking work on our nifi flow migration

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Chaffelson/nipyapi/issues/258, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZAZODIHY4ENEFT3TJB4ELS4G4OBANCNFSM4WXTXGLQ .

Looks like that resolves the anonymous user issue but now I get this:

nipyapi.versioning.import_flow_version("5d7bc617-b702-4ba2-8168-f7bf78d3c391", encoded_flow=None, file_path='./process-groups/tmp/flow.json', flow_name=None, flow_id="c935dd5f-a012-4bbd-b7db-05a64354f84d")

Traceback (most recent call last): File "", line 1, in File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\versioning.py", line 685, in import_flow_version nipyapi.utils.validate_parameters_versioning_support() File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\utils.py", line 525, in validate_parameters_versioning_support '0.6', service='registry', bool_response=True) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\utils.py", line 544, in enforce_min_ver if check_version(min_version, service=service) == 1: File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\utils.py", line 496, in check_version auth_settings=['tokenAuth', 'Authorization'] File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\api_client.py", line 332, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\api_client.py", line 153, in __call_api _request_timeout=_request_timeout) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\api_client.py", line 355, in request headers=headers) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\rest.py", line 233, in GET query_params=query_params) File "C:\Users\wzrbxs\AppData\Roaming\Python\Python37\site-packages\nipyapi\registry\rest.py", line 224, in request raise ApiException(http_resp=r) nipyapi.registry.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 28 Jan 2021 22:09:09 GMT', 'Content-Type': 'text/plain;charset=iso-8859-1', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'Strict-Transport-Security': 'max-age=31540000 ; includeSubDomains', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Security-Policy': "frame-ancestors 'self'", 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(9.4.19.v20190610)'}) HTTP response body: b'Access is denied due to: Unable to view swagger. Contact the system administrator.\n'

Update: turns out our Registry is version 0.6.0 not 0.8.0. Also it appears there is no registry-0.6.0.json in the api_defs folder of nipy, could this be causing this issue?

So the problem is most likely that you do not have permission to download the swagger.json from the Registry, which is a new requirement. The way I've coded it at the moment it'll throw an error in that case. Perhaps it would be more pythonic to attempt the call and handle the failure if your version of registry is too old to support that feature. Otherwise you can grant your user the swagger read authorization on Registry and it should work

@crhicko as @Chaffelson mentioned you are most likely missing "/swagger" read policy in Registry for user with which you are trying to make a call

Now, you cannot create that policy through UI. I happened to have the same issue you are meeting with and don't know whether you still have the same issue. So, I'll just post solution so someone else might find his way if he/she is lost.

Best approach would be creating that policy through rest API if you can but if you cannot manual steps would be:

Find user ID (user to which you want to give access to swagger) in authorizations.xml file or group ID (would be even better to respect role base access if you follow one)
Stop NiFi-Registry
Find a swagger policy line in authorizations.xml
Add extra <user identifier="<your-userid"/> or <group identifier="<your-groupid"/> line for that already created policy
Save and restart Registry

You can even try it to do through nipyapi if you want so. Here is the example which is giving user "nobel" read access on bucket: https://github.com/Chaffelson/nipyapi/blob/5480af8fe8c6b470249837835cb1a067abb6678e/nipyapi/demo/secure_connection.py#L138

you can copy/paste part of code needed for creating policy and have:

   all_buckets_access_policies = [
        ("read", "/swagger")
    ]

instead of what's written in file above, haven't tried it but it should work.

Hope it helps..

To provide clarity about this, I'm not sure that requiring you provide access to the swagger to exactly determine the version is the right solution. Possibly there is a way to infer the version by making some call which has changed behavior in Registry 0.6.0+, though I'm not aware of one. Possibly also we could simply issue a warning to the user that we cannot determine Registry version, so proceed at your own risk - and put this behind a 'strict' flag which defaults to False. If you have other suggestions I am more than happy to discuss them.

On Wed, Feb 3, 2021 at 7:27 PM Ivan Majnarić notifications@github.com wrote:

@crhicko https://github.com/crhicko as @Chaffelson https://github.com/Chaffelson mentioned you are most likely missing "/swagger" read policy in Registry for user with which you are trying to make a call

Now, you cannot create that policy through UI. I happened to have the same issue you are meeting with and don't know whether you still have the same issue. So, I'll just post solution so someone else might find his way if he/she is lost.

Best approach would be creating that policy through rest API if you can but if you cannot manual steps would be:

Find user ID (user to which you want to give access to swagger) in authorizations.xml file or group ID (would be even better to respect role base access if you follow one)

Stop NiFi-Registry

Find a swagger policy line in authorizations.xml

Add extra or <group identifier="<your-groupid"/> line for that already created policy

Save and restart Registry

You can even try it to do through nipyapi if you want so. Here is the example which is giving user "nobel" read access on bucket: https://github.com/Chaffelson/nipyapi/blob/5480af8fe8c6b470249837835cb1a067abb6678e/nipyapi/demo/secure_connection.py#L138

you can copy/paste part of code needed for creating policy and have:

all_buckets_access_policies = [ ("read", "/swagger") ]

instead of what's written in file above, haven't tried it but it should work.

Hope it helps..

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Chaffelson/nipyapi/issues/258#issuecomment-772746153, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZAZODIMMSAHPWVNRWQN6TS5GM7ZANCNFSM4WXTXGLQ .

I've the same issue how to fix? also I've this issue. in wait_to_complete raise ValueError("Timed Out waiting for {0} to complete".format( ValueError: Timed Out waiting for is_endpoint_up to complete

This should be resolved in new versions which use the AboutAPI to get the NiFi-Registry version, and if that fails try to get the swagger.json, and if that fails now issue a warning and use a default version.

Chaffelson / nipyapi