Open anden-akkio opened 1 year ago
I know there's a bug that was introduced somewhere in meltano that doesn't show the pip installation logs for some reason. I'm not sure if it was 3.x that caused this but https://github.com/meltano/meltano/pull/8145 should have addressed it.
You could try running the add command with a 2.x version of meltano as well or run with the version that's on main.
I'm pretty sure this is failing during the installation of tap-mysql
so seeing those pip logs is important here!
Attempted install on 2.x.x and it's indeed a pip installation failure:
Extractor 'tap-mysql' could not be installed: failed to install plugin 'tap-mysql'.
Running command git clone --filter=blob:none --quiet https://github.com/MeltanoLabs/tap-mysql.git /private/var/folders/kt/k_mss0s12y5dkzvyfbbkcrsm0000gn/T/pip-req-build-uks9lc5u
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [22 lines of output]
Trying pkg-config --exists mysqlclient
Command 'pkg-config --exists mysqlclient' returned non-zero exit status 1.
Trying pkg-config --exists mariadb
Command 'pkg-config --exists mariadb' returned non-zero exit status 1.
Traceback (most recent call last):
File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mysql/venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mysql/venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/Users/andenacitelli/meltano-service/src/meltano/.meltano/extractors/tap-mysql/venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/private/var/folders/kt/k_mss0s12y5dkzvyfbbkcrsm0000gn/T/pip-build-env-7uf0tg37/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/private/var/folders/kt/k_mss0s12y5dkzvyfbbkcrsm0000gn/T/pip-build-env-7uf0tg37/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
self.run_setup()
File "/private/var/folders/kt/k_mss0s12y5dkzvyfbbkcrsm0000gn/T/pip-build-env-7uf0tg37/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
exec(code, locals())
File "<string>", line 154, in <module>
File "<string>", line 48, in get_config_posix
File "<string>", line 27, in find_package_name
Exception: Can not find valid pkg-config name.
Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Looks like it wants mysql
or mariadb
to be present. I see there are some instructions in the README that I missed as to this end. Our deployment is on Linux so we could theoretically use APT, but we do our work on MacOS so I'm not really sure whether an equivalent exists. Advice?
Worth noting that this isn't a blocker because the singer-io
implementation does the job sufficiently for now with no extra config. Happy to see this through to improve Meltano though, as we'd like to switch to a more modern connector w/ more consistent updates!
@visch Edited some more information in you may want to take a look at, appreciate your help
@anden-akkio While we could use the pure python implementation here it's slower and not the recommended method according to SQLAlchemy. The steps here https://www.bytebase.com/blog/how-to-install-mysql-client-on-mac-ubuntu-centos-windows/ look find from a quick google search , we could add a reference for MacOS installation instructions to the Readme. If you could get the steps and add that it'd be great!
I am also running into an issue on Mac whilst trying to get this to work.
❯ meltano add extractor tap-mysql --variant meltanolabs
Extractor 'tap-mysql' already exists in your Meltano project
To switch from the current 'transferwise' variant to 'meltanolabs':
1. Update `variant` and `pip_url` in your `meltano.yml` project file:
name: tap-mysql
variant: meltanolabs
pip_url: git+https://github.com/MeltanoLabs/tap-mysql.git
2. Reinstall the plugin:
meltano install extractor tap-mysql
3. Check if the configuration is still valid (and make changes until it is):
meltano config tap-mysql list
To learn more, visit https://docs.meltano.com/guide/plugin-management#switching-from-one-variant-to-another
Alternatively, to keep the existing 'tap-mysql' with variant 'meltanolabs', add variant 'meltanolabs' as a separate plugin with its own unique name:
meltano add extractor tap-mysql--meltanolabs --inherit-from tap-mysql --variant meltanolabs
To learn more, visit https://docs.meltano.com/guide/plugin-management#multiple-variants
2024-07-06T18:29:47.382967Z [info ] Installing extractor 'tap-mysql'
2024-07-06T18:29:51.396660Z [info ] Logged pip install output to /Users/sigex/workdir/meltano-first-tutorial/meltano-project-github/.meltano/logs/pip/extractors/tap-mysql/pip.log
2024-07-06T18:29:51.397011Z [error ] Extractor 'tap-mysql' could not be installed: Failed to install plugin 'tap-mysql'.
2024-07-06T18:29:51.397064Z [info ] Collecting pipelinewise-tap-mysql
Using cached pipelinewise_tap_mysql-1.6.0-py3-none-any.whl.metadata (23 kB)
Collecting pendulum==2.1.2 (from pipelinewise-tap-mysql)
Using cached pendulum-2.1.2.tar.gz (81 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pipelinewise-singer-python==1.* (from pipelinewise-tap-mysql)
Using cached pipelinewise_singer_python-1.3.0-py3-none-any.whl.metadata (2.6 kB)
Collecting mysql-replication==0.43 (from pipelinewise-tap-mysql)
Using cached mysql_replication-0.43.0-py3-none-any.whl
Collecting PyMySQL==1.1.* (from pipelinewise-tap-mysql)
Using cached PyMySQL-1.1.1-py3-none-any.whl.metadata (4.4 kB)
Collecting plpygis==0.2.1 (from pipelinewise-tap-mysql)
Using cached plpygis-0.2.1-py3-none-any.whl.metadata (2.3 kB)
Collecting tzlocal==2.1 (from pipelinewise-tap-mysql)
Using cached tzlocal-2.1-py2.py3-none-any.whl.metadata (8.2 kB)
Collecting python-dateutil<3.0,>=2.6 (from pendulum==2.1.2->pipelinewise-tap-mysql)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytzdata>=2020.1 (from pendulum==2.1.2->pipelinewise-tap-mysql)
Using cached pytzdata-2020.1-py2.py3-none-any.whl.metadata (2.3 kB)
Collecting pytz<2021.0 (from pipelinewise-singer-python==1.*->pipelinewise-tap-mysql)
Using cached pytz-2020.5-py2.py3-none-any.whl.metadata (21 kB)
Collecting jsonschema==3.2.0 (from pipelinewise-singer-python==1.*->pipelinewise-tap-mysql)
Using cached jsonschema-3.2.0-py2.py3-none-any.whl.metadata (7.8 kB)
Collecting simplejson==3.17.2 (from pipelinewise-singer-python==1.*->pipelinewise-tap-mysql)
Using cached simplejson-3.17.2-cp312-cp312-macosx_13_0_arm64.whl
Collecting backoff==1.10.0 (from pipelinewise-singer-python==1.*->pipelinewise-tap-mysql)
Using cached backoff-1.10.0-py2.py3-none-any.whl.metadata (12 kB)
Collecting ciso8601 (from pipelinewise-singer-python==1.*->pipelinewise-tap-mysql)
Using cached ciso8601-2.3.1-cp312-cp312-macosx_11_0_arm64.whl.metadata (25 kB)
Collecting attrs>=17.4.0 (from jsonschema==3.2.0->pipelinewise-singer-python==1.*->pipelinewise-tap-mysql)
Using cached attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
Collecting pyrsistent>=0.14.0 (from jsonschema==3.2.0->pipelinewise-singer-python==1.*->pipelinewise-tap-mysql)
Using cached pyrsistent-0.20.0-cp312-cp312-macosx_10_9_universal2.whl.metadata (27 kB)
Collecting setuptools (from jsonschema==3.2.0->pipelinewise-singer-python==1.*->pipelinewise-tap-mysql)
Using cached setuptools-70.2.0-py3-none-any.whl.metadata (5.8 kB)
Collecting six>=1.11.0 (from jsonschema==3.2.0->pipelinewise-singer-python==1.*->pipelinewise-tap-mysql)
Using cached six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Using cached pipelinewise_tap_mysql-1.6.0-py3-none-any.whl (42 kB)
Using cached pipelinewise_singer_python-1.3.0-py3-none-any.whl (24 kB)
Using cached plpygis-0.2.1-py3-none-any.whl (21 kB)
Using cached PyMySQL-1.1.1-py3-none-any.whl (44 kB)
Using cached tzlocal-2.1-py2.py3-none-any.whl (16 kB)
Using cached backoff-1.10.0-py2.py3-none-any.whl (31 kB)
Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached pytz-2020.5-py2.py3-none-any.whl (510 kB)
Using cached pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
Using cached ciso8601-2.3.1-cp312-cp312-macosx_11_0_arm64.whl (15 kB)
Using cached attrs-23.2.0-py3-none-any.whl (60 kB)
Using cached pyrsistent-0.20.0-cp312-cp312-macosx_10_9_universal2.whl (83 kB)
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Using cached setuptools-70.2.0-py3-none-any.whl (930 kB)
Building wheels for collected packages: pendulum
Building wheel for pendulum (pyproject.toml): started
error: subprocess-exited-with-error
× Building wheel for pendulum (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Building wheel for pendulum (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for pendulum
Failed to build pendulum
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (pendulum)
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.
Failed to install plugin(s)
I get the same issue with meltano add extractor tap-mysql
and meltano add extractor tap-mysql --variant meltanolabs
I can confirm that adding the singer io one does go through.
❯ meltano add extractor tap-mysql --variant singer-io
Added extractor 'tap-mysql' to your project
Variant: singer-io
Repository: https://github.com/singer-io/tap-mysql
Documentation: https://hub.meltano.com/extractors/tap-mysql--singer-io
2024-07-06T18:31:28.562833Z [info ] Installing extractor 'tap-mysql'
2024-07-06T18:31:34.006730Z [info ] Installed extractor 'tap-mysql'
To learn more about extractor 'tap-mysql', visit https://hub.meltano.com/extractors/tap-mysql--singer-io
But I then run into another issue
❯ meltano config tap-mysql set host localhost
meltano config tap-mysql set port 3306
meltano config tap-mysql set user root
meltano config tap-mysql set password root
meltano config tap-mysql set database mydatabase
2024-07-06T18:32:22.712434Z [info ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
/Users/sigex/.local/pipx/venvs/meltano/lib/python3.12/site-packages/meltano/cli/interactive/config.py:404: RuntimeWarning: Unknown setting 'host'
value, metadata = settings.set_with_metadata(
Extractor 'tap-mysql' setting 'host' was set in `meltano.yml`: 'localhost'
2024-07-06T18:32:23.212641Z [info ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
/Users/sigex/.local/pipx/venvs/meltano/lib/python3.12/site-packages/meltano/cli/interactive/config.py:404: RuntimeWarning: Unknown setting 'port'
value, metadata = settings.set_with_metadata(
Extractor 'tap-mysql' setting 'port' was set in `meltano.yml`: '3306'
2024-07-06T18:32:23.703357Z [info ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
/Users/sigex/.local/pipx/venvs/meltano/lib/python3.12/site-packages/meltano/cli/interactive/config.py:404: RuntimeWarning: Unknown setting 'user'
value, metadata = settings.set_with_metadata(
Extractor 'tap-mysql' setting 'user' was set in `meltano.yml`: 'root'
2024-07-06T18:32:24.183287Z [info ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
/Users/sigex/.local/pipx/venvs/meltano/lib/python3.12/site-packages/meltano/cli/interactive/config.py:404: RuntimeWarning: Unknown setting 'password'
value, metadata = settings.set_with_metadata(
Extractor 'tap-mysql' setting 'password' was set in `meltano.yml`: 'root'
2024-07-06T18:32:24.698580Z [info ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
/Users/sigex/.local/pipx/venvs/meltano/lib/python3.12/site-packages/meltano/cli/interactive/config.py:404: RuntimeWarning: Unknown setting 'database'
value, metadata = settings.set_with_metadata(
Extractor 'tap-mysql' setting 'database' was set in `meltano.yml`: 'mydatabase'
❯ meltano config tap-mysql test
2024-07-06T18:32:33.918991Z [info ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.
Plugin configuration is invalid
Catalog discovery failed: command ['/Users/sigex/workdir/meltano-first-tutorial/meltano-project-github/.meltano/extractors/tap-mysql/venv/bin/tap-mysql', '--config', '/Users/sigex/workdir/meltano-first-tutorial/meltano-project-github/.meltano/run/tap-mysql/tap.b470d9dc-bed2-4d11-bee6-e7329e2fd72c.config.json', '--discover'] returned 1 with stderr:
Traceback (most recent call last):
File "/Users/sigex/workdir/meltano-first-tutorial/meltano-project-github/.meltano/extractors/tap-mysql/venv/bin/tap-mysql", line 5, in <module>
from tap_mysql import main
File "/Users/sigex/workdir/meltano-first-tutorial/meltano-project-github/.meltano/extractors/tap-mysql/venv/lib/python3.12/site-packages/tap_mysql/__init__.py", line 24, in <module>
import tap_mysql.sync_strategies.binlog as binlog
File "/Users/sigex/workdir/meltano-first-tutorial/meltano-project-github/.meltano/extractors/tap-mysql/venv/lib/python3.12/site-packages/tap_mysql/sync_strategies/binlog.py", line 21, in <module>
from tap_mysql.connection import connect_with_backoff, MySQLConnection, make_connection_wrapper
File "/Users/sigex/workdir/meltano-first-tutorial/meltano-project-github/.meltano/extractors/tap-mysql/venv/lib/python3.12/site-packages/tap_mysql/connection.py", line 17, in <module>
match_hostname = ssl.match_hostname
^^^^^^^^^^^^^^^^^^
AttributeError: module 'ssl' has no attribute 'match_hostname'
Overview
Attempting to install this via
meltano add
but am running into an error that is extremely vague to the point that I'm not really sure where to start debugging.Other extractors install fine. For example, just ran
meltano --log-level=debug add extractor tap-mysql --variant singer-io
and it went through fine.CLI Output
meltano.yml
What I've Tried
.meltano/
and try againmeltano install --clean