Fix: dagster._check.CheckError: Value in dict mismatches expected type for key MELTANO_CLI_LOG_CONFIG

[jpuris]

Hi!

When going through a blog post done by Fraser Marlow Orchestrate Meltano Jobs with Dagster, one would run into problems at section Option 1: import jobs from Meltano.

The error presented in the Dagster UI as well as stdout inside terminal running the dagster dev will output following

2023-05-15 20:13:50 +0200 - dagster - ERROR - my_meltano_job - 1ee48ebd-68c7-4a80-ad6e-d6e8391f97ed - 77136 - tap_github_target_postgres - STEP_FAILURE - Execution of step "tap_github_target_postgres" failed.

dagster._core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op "tap_github_target_postgres"::

dagster._check.CheckError: Value in dict mismatches expected type for key MELTANO_CLI_LOG_CONFIG. Expected value of type <class 'str'>. Got value /Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster_meltano/logging.yaml of type <class 'pathlib.PosixPath'>.

Stack Trace:
  File "/Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster/_core/execution/plan/utils.py", line 54, in op_execution_error_boundary
    yield
  File "/Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster/_utils/__init__.py", line 445, in iterate_with_context
    next_output = next(iterator)
  File "/Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 124, in _coerce_op_compute_fn_to_iterator
    result = invoke_compute_fn(
  File "/Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster/_core/execution/plan/compute_generator.py", line 118, in invoke_compute_fn
    return fn(context, **args_to_pass) if context_arg_provided else fn(**args_to_pass)
  File "/Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster_meltano/ops.py", line 96, in dagster_op
    output = meltano_resource.execute_command(f"{command}", env, context.log)
  File "/Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster_meltano/meltano_resource.py", line 60, in execute_command
    output, exit_code = execute_shell_command(
  File "/Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster_shell/utils.py", line 161, in execute
    return execute_script_file(
  File "/Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster_shell/utils.py", line 63, in execute_script_file
    env = check.opt_nullable_dict_param(env, "env", key_type=str, value_type=str)
  File "/Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster/_check/__init__.py", line 315, in opt_nullable_dict_param
    return _check_mapping_entries(obj, key_type, value_type, mapping_type=dict)
  File "/Users/janispuris/dag-melt/dagster/dag-venv/lib/python3.10/site-packages/dagster/_check/__init__.py", line 1825, in _check_mapping_entries
    raise CheckError(
E0515 20:18:17.401159000 6243414016 fork_posix.cc:76]                  Other threads are currently calling into gRPC, skipping fork() handlers
...

I believe the problems comes from the fact that at some point Dagster had introduced type checking, which then breaks as it would expect a Dict[str, str] object passed as env, but instead gets a Dict with a Pathlib's object in it.

This PR will convert the offending key's value i.e. MELTANO_CLI_LOG_CONFIG into string before returning the dictionary for use downstream.

[jpuris]

I did not try to run any other scenarios other than Fraser Marlow's Orchestrate Meltano Jobs with Dagster first "Option" described in the blog post.

I do believe this problem affects most other use cases involving running Meltano jobs as it would often (or always?) involve the offending "default_env" property from MeltanoResource 🤷

[JulesHuisman]

Hi Janis,

Thanks for fixing the issue!

In the future I will add some more automated tests that run on newer Dagster versions to make sure it doesn't break in the future.

[jlloyd-widen]

I just got this error in my stage environment (AWS ECS) but I don't get it in my local environment (docker-compose).

dagster._check.CheckError: Value in dict mismatches expected type for key MELTANO_CLI_LOG_CONFIG. Expected value of type <class 'str'>. Got value /opt/venv/lib/python3.8/site-packages/dagster_meltano/logging.yaml of type <class 'pathlib.PosixPath'>

I'm running dagster==1.4.6, meltano==2.18.0, dagster-meltano==1.3.1. any thoughts on how to resolve?

[JulesHuisman]

We made a change to the loading of the environment variables in the latest release. That might have something to do with it?

https://github.com/quantile-development/dagster-meltano/compare/v1.3.0...v1.3.1

[jlloyd-widen]

Possibly, though that code change seems benign. It looks like I had to actually pass MELTANO_CLI_LOG_CONFIG into the env config to get it to work ... Not a big deal I suppose, but odd that it's not working as expected.