Tap fails when ssl is enabled but sslrootcert doesn't exist

[jcbmllgn]

How to reproduce

I have the following setup in my meltano.yml file:

  extractors:
  - name: tap-postgres-arc-incremental
    inherit_from: tap-postgres
    variant: meltanolabs
    pip_url: git+https://github.com/MeltanoLabs/tap-postgres.git
    config:
      host: ${ARC_DB_HOST}
      user: ${ARC_DB_USER}
      port: ${ARC_DB_PORT}
      password: ${ARC_DB_PASSWORD}
      database: ${ARC_DB_NAME}
      default_replication_method: INCREMENTAL
      filter_schemas:
      - public
      ssl_enable: true
      ssl_mode: require

and I'm getting this error:

Cannot list the selected attributes: Catalog discovery failed: command ['/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', '/Users/jacob/code/pairteam/analytics-etl/.meltano/run/tap-postgres-arc-incremental/tap.bc22fdb4-0d0b-45b1-98c8-51dbb324bda4.config.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(TapPostgres.cli())
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/click/core.py", line 2362, in process_value
    value = self.callback(ctx, self, value)
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 529, in cb_discover
    tap.run_discovery()
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 289, in run_discovery
    catalog_text = self.catalog_json_text
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 309, in catalog_json_text
    return json.dumps(self.catalog_dict, indent=2)
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_postgres/tap.py", line 512, in catalog_dict
    result["streams"].extend(self.connector.discover_catalog_entries())
  File "/Users/jacob/.pyenv/versions/3.10.11/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_postgres/tap.py", line 408, in connector
    url = make_url(self.get_sqlalchemy_url(config=self.config))
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_postgres/tap.py", line 329, in get_sqlalchemy_url
    query=self.get_sqlalchemy_query(config=config),
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_postgres/tap.py", line 348, in get_sqlalchemy_query
    query["sslrootcert"] = self.filepath_or_certificate(
  File "/Users/jacob/code/pairteam/analytics-etl/.meltano/extractors/tap-postgres/venv/lib/python3.10/site-packages/tap_postgres/tap.py", line 394, in filepath_or_certificate
    with open(alternative_name, "wb") as alternative_file:
FileNotFoundError: [Errno 2] No such file or directory: '.secrets/root.crt'

I have no cert at this location .secrets/root.crt but this shouldn't cause the tap to fail.

According to a slack conversation with @visch the issue is that "there's a bug with the require ssl_mode as we shouldn't require a cert authority to be provided in that case. The tests just optimized for the most secure ssl method" and Update the code here to not send an sslrootcert for different ssl modes.

[ondramie]

what's the solution to this? download the pem and put in the .secrets folder?

[visch]

what's the solution to this? download the pem and put in the .secrets folder?

We'd love a PR to address this for this usecase. I think he had to point the ssl cert configuration to a random file so there was something to read.

[jcbmllgn]

Like @visch said, I created an empty file and pointed the ssl cert configuration to it. Janky and not ideal.. but it is technically working.