search

GoogleCloudPlatform / professional-services-data-validator

Utility to compare data between homogeneous or heterogeneous environments to ensure source and target tables match
Apache License 2.0
402 stars 115 forks source link

NotImplementedError: Could not find signature for dtype: <MySQLDialect_pymysql, JSON> #687

Closed ghost closed 1 year ago

ghost commented 1 year ago

I am trying to perform validation between MySQL and BigQuery for simple table it works. But for 1 table I am getting below error. I have referred to issue #409 but the solution is not working for me. Please help.

`Traceback (most recent call last): File "path/venv/lib/python3.8/site-packages/multipledispatch/dispatcher.py", line 269, in call func = self._cache[types] KeyError: (<class 'sqlalchemy.dialects.mysql.pymysql.MySQLDialect_pymysql'>, <class 'sqlalchemy.dialects.mysql.json.JSON'>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "path/venv/bin/data-validation", line 11, in load_entry_point('google-pso-data-validator==2.8.0', 'console_scripts', 'data-validation')() File "path/venv/lib/python3.8/site-packages/data_validation/main.py", line 586, in main validate(args) File "path/venv/lib/python3.8/site-packages/data_validation/main.py", line 562, in validate run(args) File "path/venv/lib/python3.8/site-packages/data_validation/main.py", line 522, in run run_validations(args, config_managers) File "path/venv/lib/python3.8/site-packages/data_validation/main.py", line 478, in run_validations run_validation(config_manager, verbose=args.verbose) File "path/venv/lib/python3.8/site-packages/data_validation/main.py", line 467, in run_validation validator.execute() File "path/venv/lib/python3.8/site-packages/data_validation/data_validation.py", line 97, in execute result_df = self._execute_validation( File "path/venv/lib/python3.8/site-packages/data_validation/data_validation.py", line 292, in _execute_validation target_query = validation_builder.get_target_query() File "path/venv/lib/python3.8/site-packages/data_validation/validation_builder.py", line 383, in get_target_query table = self.config_manager.get_target_ibis_table() File "path/venv/lib/python3.8/site-packages/data_validation/config_manager.py", line 344, in get_target_ibis_table self._target_ibis_table = clients.get_ibis_table( File "path/venv/lib/python3.8/site-packages/data_validation/clients.py", line 147, in get_ibis_table return client.table(table_name, database=schema_name) File "path/venv/lib/python3.8/site-packages/ibis/backends/mysql/client.py", line 205, in table node = self.table_class(alch_table, self, self._schemas.get(name)) File "path/venv/lib/python3.8/site-packages/ibis/backends/base_sqlalchemy/alchemy.py", line 1024, in init schema = sch.infer(table, schema=schema) File "path/venv/lib/python3.8/site-packages/multipledispatch/dispatcher.py", line 278, in call return func(*args, **kwargs) File "path/venv/lib/python3.8/site-packages/ibis/backends/base_sqlalchemy/alchemy.py", line 298, in schema_from_table dtype = dt.dtype( File "path/venv/lib/python3.8/site-packages/multipledispatch/dispatcher.py", line 273, in call raise NotImplementedError( NotImplementedError: Could not find signature for dtype: <MySQLDialect_pymysql, JSON>`

dorangao commented 1 year ago

can you help prioritize this?

helensilva14 commented 1 year ago

Hi! Just tested with our latest version (v4.2.0) and we do support JSON data type in MySQL 8.0 now. Please check evidence below:

CREATE TABLE issue687_json_type_support (jdoc JSON); 
INSERT INTO issue687_json_type_support VALUES('{"key1": "value1", "key2": "value2"}');

DVT command: data-validation validate schema -sc mysql -tc mysql -tbls guestbook.issue687_json_type_support

image