Unable to set using_ssl via YAML and other properties via command line

[ChrisSamo632]

Describe the bug Due to #61, I attempted to specify using_ssl: True in a config.yaml file along with other settings as command line parameters.

To Reproduce Steps to reproduce the behavior:

• config.yaml like (per #61, I've tried other variants of the using_ssl field)

  configuration_version : 1
  using_ssl: True

• execute dotifi from the CLI/bash script and provide one or more parameters
• check the parameter settings used by dotifi (e.g. by viewing the output of --verbose mode; note that using_ssl: no)

  2020-09-11 18:42:36,861 root.process +203: DEBUG    [82] Configuration:
  with_conf_file: /usr/src/config/config.yaml
  output_dot_file: /data/root.dot
  output_graphviz_format: png
  output_graphviz_program: dot
  output_graphviz_file: /data/root.png
  depth: '-1'
  nifi_url: https://nifi-cs:8080/nifi-api
  using_ssl: no
  using_user_pw: no
  ca_file: /usr/src/certs/nifi-admin/nifi-cert.pem
  client_cert_file: /usr/src/certs/nifi-admin/nifi-admin-cert.pem
  client_key_file: /usr/src/certs/nifi-admin/nifi-admin-key.pem
  client_key_password: XXX
  verbose: yes
  generate_mock_data: no
  mock_data_file: mock_output.json
  configuration_version: 1

Expected behavior Parameters from a config YAML and command line are combined (I guess you'd need to decide which take precedence, I'd maybe suggest those from the CLI?).

Desktop (please complete the following information):

• OS: Ubuntu VERSION: 20.04
• NIFI: 1.11.4
• Python: 3.8
• dotifi: 0.1.2

Additional context Unfortunately, dotifi seems to either ignore the YAML (I checked the file reference I provided was correct and contained the contents I expected) or the ArgsParser defaults (in dotifi.py) are overriding the values from the YAML.

Due to being hosted in Kubernetes, I don't necessarily know all of the settings I'd need to provide to dotifi until the Pod/container are started within the Kubernetes namespace, therefore at least some command line parameters seem necessary (#61 prevents my being able to specify all needed parameters as part of a bash script).

This leads to dotifi (via NiPyApi) attempting to connect to my https protected NiFi without SSL enabled:

2020-09-11 18:42:36,861 nipyapi.utils.set_endpoint +296: INFO [82] Called set_endpoint with args {'endpoint_url': 'https://nifi-cs:8080/nifi-api', 'ssl': False, 'login': False}

and ultimately:

2020-09-11 18:42:36,990 root.process +220: ERROR    [82] HTTPSConnectionPool(host='nifi-cs', port=8080): Max retries exceeded with url: /nifi-api/flow/process-groups/root/status (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)')))
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 362, in connect
    self.sock = ssl_wrap_socket(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 384, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1123)

[ChrisSamo632]

After #64 I've tried with setting other config via YAML while other parameters are set on the command line and it seems to work for those (for example, output_graphviz_format: png in the YAML and --using-ssl on the command line along with the certificate files).

So it might have just been the flags such as using_ssl that was wrong, or I could have been doing something wrong previously with my YAML file.

If you can't reproduce this issue, happy for it to be closed.

[ottobackwards]

I want to try it and see if I need to document a difference between the cli and the yaml, such as "if you set through the command line, just the flag is required, but in yaml you need to put true or false"