search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.7k stars 1.48k forks source link

CLI repo-loading errors are less legible than before #2772

Closed sryza closed 4 years ago

sryza commented 4 years ago

I had a bug in a python module that I was trying to load a repo from to execute on the CLI. The actual error that's useful is buried inside the message of the DagsterIPCProtocolError, difficult to understand because the "\n"s don't render as newlines.

Traceback (most recent call last):
  File "/Users/sryza/dagster/python_modules/dagster/dagster/api/utils.py", line 11, in execute_command_in_subprocess
    subprocess.check_output(parts, stderr=subprocess.STDOUT)
  File "/Users/sryza/.pyenv/versions/3.6.8/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/Users/sryza/.pyenv/versions/3.6.8/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/bin/python3.6', '-m', 'dagster', 'api', 'list_repositories', '/var/folders/df/2_jxd7dx073273d_mpywh4080000gn/T/tmpf_t93t_j', '/var/folders/df/2_jxd7dx073273d_mpywh4080000gn/T/tmpyyx3_gjt']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sryza/.pyenv/versions/dagster-3.6.8/bin/dagster", line 11, in <module>
    load_entry_point('dagster', 'console_scripts', 'dagster')()
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/__init__.py", line 38, in main
    cli(obj={})  # pylint:disable=E1123
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/pipeline.py", line 262, in pipeline_execute_command
    return _logged_pipeline_execute_command(config, preset, mode, DagsterInstance.get(), kwargs)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/core/telemetry.py", line 89, in wrap
    result = f(*args, **kwargs)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/pipeline.py", line 290, in _logged_pipeline_execute_command
    result = execute_execute_command(env, kwargs, mode, tags)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/pipeline.py", line 297, in execute_execute_command
    external_pipeline = get_external_pipeline_from_kwargs(cli_args, instance)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 404, in get_external_pipeline_from_kwargs
    external_repo = get_external_repository_from_kwargs(kwargs, instance)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 367, in get_external_repository_from_kwargs
    repo_location = get_repository_location_from_kwargs(kwargs, instance)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 335, in get_repository_location_from_kwargs
    workspace = get_workspace_from_kwargs(kwargs, instance)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 198, in get_workspace_from_kwargs
    return workspace_from_load_target(created_workspace_load_target(kwargs), instance)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 168, in workspace_from_load_target
    user_process_api=python_user_process_api,
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/load.py", line 253, in location_handle_from_python_file
    attribute=attribute,
  File "/Users/sryza/dagster/python_modules/dagster/dagster/api/list_repositories.py", line 17, in sync_list_repositories
    attribute=attribute,
  File "/Users/sryza/dagster/python_modules/dagster/dagster/api/utils.py", line 32, in execute_unary_api_cli_command
    execute_command_in_subprocess(parts)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/api/utils.py", line 14, in execute_command_in_subprocess
    "Error when executing API command {cmd}: {output}".format(cmd=e.cmd, output=e.output)
dagster.serdes.ipc.DagsterIPCProtocolError: Error when executing API command ['/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/bin/python3.6', '-m', 'dagster', 'api', 'list_repositories', '/var/folders/df/2_jxd7dx073273d_mpywh4080000gn/T/tmpf_t93t_j', '/var/folders/df/2_jxd7dx073273d_mpywh4080000gn/T/tmpyyx3_gjt']: b'/Users/sryza/dagster/python_modules/libraries/dagster-pandas/dagster_pandas/data_frame.py:190: UserWarning: Using create_dagster_pandas_dataframe_type for dataframe types is deprecated,\n     and is planned to be removed in a future version (tentatively 0.10.0).\n     Please use create_structured_dataframe_type instead.\n  Please use create_structured_dataframe_type instead."""\nTraceback (most recent call last):\n  File "/Users/sryza/.pyenv/versions/3.6.8/lib/python3.6/runpy.py", line 193, in _run_module_as_main\n    "__main__", mod_spec)\n  File "/Users/sryza/.pyenv/versions/3.6.8/lib/python3.6/runpy.py", line 85, in _run_code\n    exec(code, run_globals)\n  File "/Users/sryza/dagster/python_modules/dagster/dagster/__main__.py", line 3, in <module>\n    main()\n  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/__init__.py", line 38, in main\n    cli(obj={})  # pylint:disable=E1123\n  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 764, in __call__\n    return self.main(*args, **kwargs)\n  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 717, in main\n    rv = self.invoke(ctx)\n  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 1137, in invoke\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\n  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 1137, in invoke\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\n  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 956, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 555, in invoke\n    return callback(*args, **kwargs)\n  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/api.py", line 115, in command\n    output = check.inst(fn(args), output_cls)\n  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/api.py", line 140, in list_repositories_command\n    loadable_targets = get_loadable_targets(python_file, module_name, working_directory, attribute)\n  File "/Users/sryza/dagster/python_modules/dagster/dagster/grpc/utils.py", line 20, in get_loadable_targets\n    else loadable_targets_from_python_file(python_file, working_directory)\n  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/autodiscovery.py", line 11, in loadable_targets_from_python_file\n    loaded_module = load_python_file(python_file, working_directory)\n  File "/Users/sryza/dagster/python_modules/dagster/dagster/core/code_pointer.py", line 88, in load_python_file\n    return import_module_from_path(module_name, python_file)\n  File "/Users/sryza/dagster/python_modules/dagster/dagster/seven/__init__.py", line 110, in import_module_from_path\n    spec.loader.exec_module(module)\n  File "<frozen importlib._bootstrap_external>", line 678, in exec_module\n  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed\n  File "examples/legacy_examples/dagster_examples/simple_lakehouse/simple_lakehouse.py", line 189, in <module>\n    from dagster_examples.simple_lakehouse.daily_temperature_high_diffs import (\n  File "/Users/sryza/dagster/examples/legacy_examples/dagster_examples/__init__.py", line 31, in <module>\n    @repository\n  File "/Users/sryza/dagster/python_modules/dagster/dagster/core/definitions/decorators/repository.py", line 225, in repository\n    return _Repository()(name)\n  File "/Users/sryza/dagster/python_modules/dagster/dagster/core/definitions/decorators/repository.py", line 23, in __call__\n    repository_definitions = fn()\n  File "/Users/sryza/dagster/examples/legacy_examples/dagster_examples/__init__.py", line 37, in legacy_examples\n    + get_lakehouse_pipelines()\n  File "/Users/sryza/dagster/examples/legacy_examples/dagster_examples/__init__.py", line 17, in get_lakehouse_pipelines\n    from dagster_examples.simple_lakehouse.pipelines import simple_lakehouse_pipeline\n  File "/Users/sryza/dagster/examples/legacy_examples/dagster_examples/simple_lakehouse/pipelines.py", line 7, in <module>\n    from dagster_examples.simple_lakehouse.simple_lakehouse import simple_lakehouse\nImportError: cannot import name \'simple_lakehouse\'\n'

What it used to look like in 0.8.5:

Traceback (most recent call last):
  File "/Users/sryza/.pyenv/versions/dagster-3.6.8/bin/dagster", line 11, in <module>
    load_entry_point('dagster', 'console_scripts', 'dagster')()
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/__init__.py", line 40, in main
    cli(obj={})  # pylint:disable=E1123
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sryza/.pyenv/versions/3.6.8/envs/dagster-3.6.8/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/core/telemetry.py", line 76, in wrap
    result = f(*args, **kwargs)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/pipeline.py", line 321, in pipeline_execute_command
    result = execute_execute_command(env, kwargs, mode, tags)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/pipeline.py", line 327, in execute_execute_command
    external_pipeline = get_external_pipeline_from_kwargs(cli_args)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 287, in get_external_pipeline_from_kwargs
    external_repo = get_external_repository_from_kwargs(kwargs)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 251, in get_external_repository_from_kwargs
    repo_location = get_repository_location_from_kwargs(kwargs)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 220, in get_repository_location_from_kwargs
    workspace = get_workspace_from_kwargs(kwargs)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 104, in get_workspace_from_kwargs
    return workspace_from_load_target(created_workspace_load_target(kwargs))
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/cli_target.py", line 93, in workspace_from_load_target
    [location_handle_from_python_file(load_target.python_file, load_target.attribute)]
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/load.py", line 145, in location_handle_from_python_file
    else loadable_targets_from_python_file(python_file)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/cli/workspace/autodiscovery.py", line 12, in loadable_targets_from_python_file
    loaded_module = load_python_file(python_file)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/core/code_pointer.py", line 73, in load_python_file
    return import_module_from_path(module_name, python_file)
  File "/Users/sryza/dagster/python_modules/dagster/dagster/seven/__init__.py", line 88, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "examples/legacy_examples/dagster_examples/simple_lakehouse/lakehouse.py", line 8, in <module>
    from lakehouse import Lakehouse, TypeStoragePolicy
ImportError: cannot import name 'Lakehouse'
gibsondan commented 4 years ago

This likely blames to https://dagster.phacility.com/D4012 from me. Obnoxiously I think the escaping may be python version-specific (need to double-check)

sryza commented 4 years ago

https://dagster.phacility.com/D4087