search

tamsanh / kedro-great

The easiest way to integrate Kedro and Great Expectations
MIT License
52 stars 14 forks source link

TypeError: Object of type 'PurePosixPath' is not JSON serializable #3

Closed dpavancini closed 4 years ago

dpavancini commented 4 years ago

Description

I'm running into an error when running the Kedro Run entrypoint with Kedro Great. It seems that Kedro is sending a PosixPath object instead of a string. I believe we can fix that adding a str() below but I don't know if there is any side effect I could run into:

if dataset_path:
    dataasset_name, _ = os.path.splitext(os.path.basename(dataset_path))
    batch_kwargs["path"] = str(dataset_path)
    batch_kwargs["data_asset_name"] = dataasset_name

Version

Kedro 0.16.2 Python 3.6.9

Error Log

Traceback (most recent call last): File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/bin/kedro", line 10, in sys.exit(main()) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/kedro/framework/cli/cli.py", line 633, in main cli_collection() File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/click/core.py", line 610, in invoke return callback(args, kwargs) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/kedro_cli.py", line 230, in run pipeline_name=pipeline, File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/kedro/framework/context/context.py", line 699, in run raise error File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/kedro/framework/context/context.py", line 691, in run run_result = runner.run(filtered_pipeline, catalog, run_id) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/kedro/runner/runner.py", line 101, in run self._run(pipeline, catalog, run_id) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/kedro/runner/sequential_runner.py", line 90, in _run run_node(node, catalog, self._is_async, run_id) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/kedro/runner/runner.py", line 213, in run_node node = _run_node_sequential(node, catalog, run_id) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/kedro/runner/runner.py", line 225, in _run_node_sequential node=node, catalog=catalog, inputs=inputs, is_async=is_async, run_id=run_id File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/pluggy/hooks.py", line 286, in call return self._hookexec(self, self.get_hookimpls(), kwargs) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/pluggy/manager.py", line 93, in _hookexec return self._inner_hookexec(hook, methods, kwargs) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/pluggy/manager.py", line 87, in firstresult=hook.spec.opts.get("firstresult") if hook.spec else False, File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/pluggy/callers.py", line 208, in _multicall return outcome.get_result() File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/pluggy/callers.py", line 80, in get_result raise ex[1].with_traceback(ex[2]) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/pluggy/callers.py", line 187, in _multicall res = hook_impl.function(args) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/kedro_great/kedro_great.py", line 81, in before_node_run self._run_validation(catalog, inputs, run_id) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/kedro_great/kedro_great.py", line 116, in _run_validation dataset_name, dataset_path, df, target_suite_name, run_id File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/kedro_great/kedro_great.py", line 182, in _run_suite "action_list_operator", [validator_dataset_batch], run_id=run_id File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/great_expectations/core/usage_statistics/usage_statistics.py", line 254, in usage_statistics_wrapped_method result = func(args, kwargs) File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/great_expectations/data_context/data_context.py", line 902, in run_validation_operator kwargs, File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/great_expectations/validation_operators/validation_operators.py", line 317, in run batch_identifier=batch.batch_id, File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/great_expectations/data_asset/data_asset.py", line 521, in batch_id return self.batch_kwargs.to_id() File "/mnt/daniel/indicium-ds/company-api-db-pipeline/venv/lib/python3.6/site-packages/great_expectations/core/id_dict.py", line 24, in to_id json.dumps(_id_dict, sort_keys=True).encode("utf-8") File "/usr/lib/python3.6/json/init.py", line 238, in dumps kw).encode(obj) File "/usr/lib/python3.6/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.6/json/encoder.py", line 180, in default o.class.name) TypeError: Object of type 'PurePosixPath' is not JSON serializable

tamsanh commented 4 years ago

@dpavancini Great catch! I don't believe there is a side effect here. Would you like to make a PR with the fix? Otherwise I can, too.

dpavancini commented 4 years ago

Sure! Just added the PR

tamsanh commented 4 years ago

Great work; looks good. I'll merge and publish @dpavancini

tamsanh commented 4 years ago

Resolved in version 0.2.5 @dpavancini. FYI @akruszewski