Registry api_client is null for imports

[shawnweeks]

• Nipyapi version: nipyapi 0.12.0
• NiFi version: 1.5 HDF 3.1.2
• NiFi-Registry version: 0.10 HDF 3.1.2
• Python version: 2.7.5
• Operating System: RedHat 7 Latest

Description

Describe what you were trying to get done, or what you would like the package to do.
Tell us what happened, what went wrong, and what you expected to happen.

Since I'm stuck on a version of NiFi Registry Prior to Git Support and Proper Import Export Capability I'm putting together some command line tools. I'm able to export and run other registry commands just fine but on the import it's failing because nipyapi.config.registry_config.api_client is None

Traceback (most recent call last): File "./import_flow", line 7, in flow = nipyapi.versioning.import_flow_version(sys.argv[1], flow_id=sys.argv[2], file_path=sys.argv[3]) File "/home/1454256952/.local/lib/python2.7/site-packages/nipyapi/versioning.py", line 685, in import_flow_version dto=dto File "/home/1454256952/.local/lib/python2.7/site-packages/nipyapi/utils.py", line 101, in load return nipyapi.config.registry_config.api_client.deserialize( AttributeError: 'NoneType' object has no attribute 'deserialize'

Here is how I'm setting up the config

import nipyapi
import subprocess

nifiToken = subprocess.check_output(['curl','-s','-k','-X POST','--negotiate','-u :','https://example.org:9091/nifi-api/access/kerberos'])
registryToken = subprocess.check_output(['curl','-s','-k','-X POST','--negotiate','-u :','https://example.org:61443/nifi-registry-api/access/token/kerberos'])

#nipyapi.config.nifi_config.debug=True
#nipyapi.config.registry_config.debug=True

nipyapi.config.nifi_config.host = 'https://example.org:9091/nifi-api'
nipyapi.config.registry_config.host = 'https://example.org:61443/nifi-registry-api'
nipyapi.security.set_service_ssl_context(service='nifi', ca_file='/home/1454256952/.pki/trust.pem', client_cert_file=None, client_key_file=None, client_key_password=None)
nipyapi.security.set_service_ssl_context(service='registry', ca_file='/home/1454256952/.pki/trust.pem', client_cert_file=None, client_key_file=None, client_key_password=None)
nipyapi.security.set_service_auth_token(token=nifiToken, token_name='tokenAuth', service='nifi')
nipyapi.security.set_service_auth_token(token=registryToken, token_name='tokenAuth', service='registry')

#!/bin/python

import config
import nipyapi
import sys

flow = nipyapi.versioning.import_flow_version(sys.argv[1], flow_id=sys.argv[2], file_path=sys.argv[3])

What I Did

To get past this I updated utils.py to actual set nipyapi.config.registry_config.api_client = ApiClient() right before the call to deserialize but I'm not sure how nipyapi.config.registry_config.api_client is supposed to get set.

Urgency

I've got a workaround so not too urgent just not sure of the implications.

[shawnweeks]

If I add this to the end of my configuration file it also works. So I'm not sure if this a bug or documentation issue.

nipyapi.config.registry_config.api_client = nipyapi.registry.api_client.ApiClient()

[Chaffelson]

Hi Shawn - I'm away on holidays at the moment but I wanted to take a minute to acknowledge your issue here even if I can't investigate it yet. To answer the immediate question - the client itself sets the api_client parameter the first time it's used, I've never had to call it directly. This allows reuse for efficiency, particularly when you've had to login, though it does mean that the registry_config.host is only referenced during initialisation, which is mildly annoying when working against multiple environments ( security.py functions should work for this ).

A couple of guesses for you to check: Could be your version of Python is causing unexpected behavior, as it is very old - can you try a newer version in a virtualenv to see if it gives a different result? You could try using utils.set_endpoint for both the NiFi and Registry and running a test transaction ( any method with a bool_response option is good for this ) to ensure it is connected as expected. This may give exactly the same error, but it would mirror my usual testing process so may give more clues.

Again my apologies that this is causing you hassle, I'll be back on deck in a week or so.

[shawnweeks]

I'm stuck on the version of Python 2.7 included with RedHat 7 which is 2.7.5. So I'll try and setup a clean VM somewhere and get a better test case put together.

[wojtek-fliposports]

I can confirm issue with Python 3.7.2. I have this error not with exporting flow but importing is affected

[shawnweeks]

Importing is what was broke for me as well. Exports run fine. I've now got Python 3.6.6 installed and I'll test it again there.

[Chaffelson]

Ah, actually I think I have an idea what this might be: Can you try using utils.set_endpoint to configure your NiFi or NiFi-Registry host, rather than setting it directly in the client?

[shawnweeks]

I'm testing this out against Registry 0.3.0 and it works both ways.

[Chaffelson]

Cool, I'm interested in your opinion in the best way to capture this problem then - should we just document the preferred use of utils.set_endpoint, or is there something we could insert in the code that would help do you think?

[shawnweeks]

Not sure anything needs to be documented. Using the way described in the documentation works. I didn't have to use utils.set_endpoint. I still need to test this against Registry 0.1.0 and see if it breaks there. Maybe the issue was accidentally fixed.

[schlichtanders]

I am facing a similar issue, just tried to use nipyapi

import nipyapi
nipyapi.config.nifi_config.host = 'http://localhost:8080/nifi-api'
nipyapi.config.registry_config.host = 'http://localhost:18080/nifi-registry-api'
nipyapi.templates.upload_template(pg_id, template_file)

of course with urls changed and parameters filled in, I get the following error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-28-c0091a11887e> in <module>
----> 1 template_entity = nipyapi.templates.upload_template(pg_id, template_file)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\nipyapi\templates.py in upload_template(pg_id, template_file)
    184     """
    185     log.info("Called upload_template against endpoint %s with args %s",
--> 186              nipyapi.config.nifi_config.api_client.host, locals())
    187     # Ensure we are receiving a valid file
    188     assert isfile(template_file) and access(template_file, R_OK), \

AttributeError: 'NoneType' object has no attribute 'host'

[David-debug]

I have the exact same issue. How did you fix it? @schlichtanders

[Chaffelson]

This appears to be caused when code refers to client information before the client has been instantiated, the client is instantiated on first request to the REST endpoint (lazy instantiation). In the case of templates/upload_template it was asking for the host endpoint before talking to it.

I recommend you add a simple connectivity check to any NiPy usage before using more advanced functions to generally avoid this issue (such as checking you can retrieve the root pg id). If you spot more sections of code where there is an explicit sequencing issue such as with upload_template, please reopen this issue or create a new one for tracking. Thanks!