search

MeltanoLabs / tap-postgres

Singer Tap for PostgreSQL
https://hub.meltano.com/extractors/tap-postgres--meltanolabs/
Other
21 stars 27 forks source link

Unexpected `ValueError: Could not determine the key type` while testing the tap #241

Open pranasziaukas opened 1 year ago

pranasziaukas commented 1 year ago

Just getting started with Meltano, and testing this Postgres tap.

So far, I have the following (redacted) meltano.yml config:

version: 1
default_environment: dev
project_id: ...
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-postgres
    variant: meltanolabs
    pip_url: git+https://github.com/MeltanoLabs/tap-postgres.git
    config:
      database: ...
      filter_schemas:
      - ...
      flattening_enabled: false
      host: ...
      ssh_tunnel:
        enable: true
        host: ...
        port: ...
        username: ...
      user: ...

As well as the following .env file

TAP_POSTGRES_PASSWORD='...'
TAP_POSTGRES_SSH_TUNNEL_PRIVATE_KEY='-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY——'

I took the SSH key file content and replaced all new lines with \n, just like in the example test case https://github.com/MeltanoLabs/tap-postgres/blob/bcc0db15be8184f7d99126fdad7b3a20f7dd91e8/tests/test_ssh_tunnel.py#L14

Furthermore, I checked out the tap code locally and ran my key against the helper method https://github.com/MeltanoLabs/tap-postgres/blob/bcc0db15be8184f7d99126fdad7b3a20f7dd91e8/tap_postgres/tap.py#L385

to be 100% sure that it's valid and detected as an RSA key type.

That's why I'm highly confused to see this when running meltano config tap-postgres test

2023-09-20T12:05:19.866936Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
Catalog discovery failed: command ['/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', '/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/run/tap-postgres/tap.598e019d-53d3-4dab-99ff-e2240091d9d6.config.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(TapPostgres.cli())
             ^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 2362, in process_value
    value = self.callback(ctx, self, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 527, in cb_discover
    tap.run_discovery()
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 287, in run_discovery
    catalog_text = self.catalog_json_text
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 307, in catalog_json_text
    return json.dumps(self.catalog_dict, indent=2)
                      ^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 475, in catalog_dict
    result["streams"].extend(self.connector.discover_catalog_entries())
                             ^^^^^^^^^^^^^^
  File "/Users/pranas/.pyenv/versions/3.11.3/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 378, in connector
    url = self.ssh_tunnel_connect(ssh_config=ssh_config, url=url)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 429, in ssh_tunnel_connect
    ssh_private_key=self.guess_key_type(ssh_config["private_key"]),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 414, in guess_key_type
    raise ValueError(errmsg)
ValueError: Could not determine the key type.

ValueError: Could not determine the key type is certainly coming from the guess_key_type method even though I have validated it as explained above. Is this a red herring? How to diagnose further?

Thanks.

visch commented 1 year ago

This is really a meltano question more than it is a tap question. From the tap perspective we need to focus just on the config file that's sent to the tap. My guess is the line feeds are mangled

pranasziaukas commented 1 year ago

Thanks for your response @visch.

I understand your perspective. I'll raise a pertinent issue in the Meltano repository. However, in my opinion, it's equally an implementation decision for the tap to manage SSH keys - which are inherently multi-line - as regular strings.

I wrote https://github.com/MeltanoLabs/tap-postgres/issues/100#issuecomment-1727697992. FWIW, I try to avoid storing raw SSH keys as environment variables precisely for the reason that newline characters are treated differently across systems and tools -- I tend to reference file paths (to SSH keys) or, if needs be, use base64 encoding to get rid of the problematic characters. Just my 2 cents.