search

akurdyukov / tap-clickhouse

Meltano SDK based tap for ClickHouse
Apache License 2.0
0 stars 2 forks source link

Port config value is being passed as an empty string on connection test #13

Open kning opened 6 months ago

kning commented 6 months ago

I have the following in my meltano.yml config

  - name: tap-clickhouse
    variant: akurdyukov
    pip_url: git+https://github.com/akurdyukov/tap-clickhouse.git
    config:
      host: $CLICKHOUSE_URL
      password: $CLICKHOUSE_PASSWORD
      port: 8443

And when I run meltano config tap-clickhouse test, I get the following:

Traceback (most recent call last):
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/bin/tap-clickhouse", line 8, in <module>
    sys.exit(TapClickHouse.cli())
             ^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 2362, in process_value
    value = self.callback(ctx, self, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 528, in cb_discover
    tap.run_discovery()
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 288, in run_discovery
    catalog_text = self.catalog_json_text
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 308, in catalog_json_text
    return json.dumps(self.catalog_dict, indent=2)
                      ^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 659, in catalog_dict
    result["streams"].extend(connector.discover_catalog_entries())
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/connectors/sql.py", line 510, in discover_catalog_entries
    engine = self._engine
             ^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/connectors/sql.py", line 309, in _engine
    self._cached_engine = self.create_engine()
                          ^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/tap_clickhouse/client.py", line 46, in create_engine
    return sqlalchemy.create_engine(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in create_engine
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 546, in create_engine
    u = _url.make_url(url)
        ^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 842, in make_url
    return _parse_url(name_or_url)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 905, in _parse_url
    return URL.create(name, **components)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 211, in create
    cls._assert_port(port),
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 221, in _assert_port
    return int(port)
           ^^^^^^^^^
ValueError: invalid literal for int() with base 10: ''
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
Catalog discovery failed: command ['/meltano_project/.meltano/extractors/tap-clickhouse/venv/bin/tap-clickhouse', '--config', '/meltano_project/.meltano/run/tap-clickhouse/tap.f748e8c9-b5ff-4f33-9268-9e43eba4fc3b.config.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/bin/tap-clickhouse", line 8, in <module>
    sys.exit(TapClickHouse.cli())
             ^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/click/core.py", line 2362, in process_value
    value = self.callback(ctx, self, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 528, in cb_discover
    tap.run_discovery()
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 288, in run_discovery
    catalog_text = self.catalog_json_text
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 308, in catalog_json_text
    return json.dumps(self.catalog_dict, indent=2)
                      ^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 659, in catalog_dict
    result["streams"].extend(connector.discover_catalog_entries())
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/connectors/sql.py", line 510, in discover_catalog_entries
    engine = self._engine
             ^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/singer_sdk/connectors/sql.py", line 309, in _engine
    self._cached_engine = self.create_engine()
                          ^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/tap_clickhouse/client.py", line 46, in create_engine
    return sqlalchemy.create_engine(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in create_engine
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 546, in create_engine
    u = _url.make_url(url)
        ^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 842, in make_url
    return _parse_url(name_or_url)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 905, in _parse_url
    return URL.create(name, **components)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 211, in create
    cls._assert_port(port),
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/meltano_project/.meltano/extractors/tap-clickhouse/venv/lib/python3.11/site-packages/sqlalchemy/engine/url.py", line 221, in _assert_port
    return int(port)
           ^^^^^^^^^
ValueError: invalid literal for int() with base 10: ''

The stack trace seems to suggest that something is happening in client.py when i builds the sqlalchemy URL. Would you be able to take a look?

akurdyukov commented 6 months ago

I tried the same with my config and it seems to work. And it seems to work in the tests. Can you please give fill tap-clickhouse config?