dagster fails to load simple code location after 1.8 upgrade

[geoHeil]

Dagster version

1.8.0

What's the issue?

The latest 1.7 (1.7.15) version of dagster can load something like this fine:

from dagster import Definitions, asset

@asset(key_prefix="playground")
def foo():
    return 1

all_assets = [foo]

defs = Definitions(
    assets=all_assets,
)

however, upgrading to 1.8.0 fails the import.

I am loading via the package workspace.yaml like:

load_from:
  - python_package:
      package_name: foo
      location_name: foo

Error is:

dagster._core.errors.DagsterInvariantViolationError: No repositories, jobs, pipelines, graphs, or asset definitions found in "other".
  File "/python3.11/site-packages/dagster/_grpc/server.py", line 411, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
                                                              ^^^^^^^^^^^^^^^^^^^
  File "/python3.11/site-packages/dagster/_grpc/server.py", line 245, in __init__
    loadable_targets = get_loadable_targets(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/python3.11/site-packages/dagster/_grpc/utils.py", line 60, in get_loadable_targets
    else loadable_targets_from_python_package(package_name, working_directory)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python3.11/site-packages/dagster/_core/workspace/autodiscovery.py", line 47, in loadable_targets_from_python_package
    return loadable_targets_from_loaded_module(module)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python3.11/site-packages/dagster/_core/workspace/autodiscovery.py", line 105, in loadable_targets_from_loaded_module
    raise DagsterInvariantViolationError(

What did you expect to happen?

the project to successfully load

How to reproduce?

see the examples above

Deployment type

Local

Deployment details

No response

Additional information

When loading without the workspace.yaml file the code location loads just fine

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[geoHeil]

I could fix this by changing the package_name to: package_name: foo.foo.

[gibsondan]

@geoHeil can you clarify what the file/package structure in the "foo" package in question when this happens? What's in the __init__.py file?

[gibsondan]

In both 1.7.15 and 1.8.0 I can reproduce that error if the __init__.py for that module is empty and doesn't explicitly import the definitions, but I haven't been successful yet at finding something there that would have regressed between 1.7.15 and 1.8.0 .

[geoHeil]

@geoHeil can you clarify what the file/package structure in the "foo" package in question when this happens? What's in the __init__.py file?

the __init__.py file is exactly like the one shared above. I think the important thing is the folder structure

├── foo
│   ├── foo
│   │   ├── __init__.py
│   │   └── __pycache__
│   │       ├── __init__.cpython-310.pyc
│   │       └── __init__.cpython-311.pyc
│   ├── foo_tests
│   │   ├── __init__.py
│   │   └── test_assets.py
│   ├── pyproject.toml
│   └── setup.cfg
├── workspace_docker.yaml
└── workspace.yaml

this is how it looks like

[gibsondan]

Thanks for sharing the folder structure - I'm still seeing identical results when I try with that folder structure on 1.7.15 and 1.8.0 though. Can you double-check that

load_from:
  - python_package:
      package_name: foo
      location_name: foo

works for you with that folder structure on 1.7.15?

[gibsondan]

(What i'm seeing on both versions is what I would expect, which is that only foo.foo works as a package name - since the __init__.py with the code in it is twofoos down in the hierarchy)

[geoHeil]

interesting - this is different for me - it works in 1.7.15 only with foo - and fails for 1.8

[geoHeil]

Given that there is a simple fix - it is ok probably if we close this one as you cannot reproduce it

[gibsondan]

Hmmm that is very strange. No need to keep digging if you don't want, but if you do - this is in the same virtual environment for both dagster versions? Is foo pip installed in some way?

[gibsondan]

(and to double-check, you're running it in the same folder on both versions? i.e. its definitely pulling its instructions from the workspace.yaml and not the pyproject.toml?)

[geoHeil]

Hmmm that is very strange. No need to keep digging if you don't want, but if you do - this is in the same virtual environment for both dagster versions? Is foo pip installed in some way?

yes, foo is pip installed - except for the dagster version change the environment is the same

[geoHeil]

(and to double-check, you're running it in the same folder on both versions? i.e. its definitely pulling its instructions from the workspace.yaml and not the pyproject.toml?)

to my best knowledge yes - is there a quick log analysis I could do to help you confirm this?

[gibsondan]

If there's a [tool.dagster] entry in the pyproject.toml, you could remove it (that would be the only way it would be able to be pulling it from somewhere other than the workspace.yaml)

[gibsondan]

Is foo installed editably ("pip install -e") or normally (just "pip install")

[gibsondan]

(maybe you could share the contents of the pyproject.toml file / setup.cfg file if relevant?)

[geoHeil]

should we jump on a quick call? I can send a zoom link maybe this is simpler than the back-and-forth?

in any case:

[tool.dagster]
module_name = "foo"

setup.cfg

[metadata]
name = foo

is part of the toml

yes editable install

[geoHeil]

When changing the toml to:

[tool.dagster]
module_name = "foo.foo"

the issue still persists - and is only fixed when changing the workspace.yml

[gibsondan]

Can’t do a call at the moment but thanks for all the back and forth.

Last question for a bit - are you switching back and forth between 1.7.15 and 1.8.0 in the same venv? Or using a different venv for each version? I just want to be sure the Dagster version is literally the only difference, because I’ve seen weird things with different versions of pip handling editable installs differently, or if foo was pip installed in one venv but not another I think that could also explain this.

[geoHeil]

I freshly create the venv with the different versions