python dataclasses seems to look in sys.modules[__module__], so all dynamic loading patterns would need to add there too (which...how do you clean up ...shrugs)

[kasium]

Describe the bug If from __future__ import annotations is used with a dataclass, alembic crashes

Expected behavior No crash

To Reproduce

1. create a migration file as below
2. run alembic heads

from __future__ import annotations

from dataclasses import dataclass

revision = "be59ab2abe34"
down_revision = "2677466a6831"

@dataclass
class Foo:
    name: str

def upgrade() -> None:
    pass

def downgrade() -> None:
    pass

Error

  File "venv/lib/python3.12/site-packages/alembic/config.py", line 624, in main
    self.run_cmd(cfg, options)
  File "venv/lib/python3.12/site-packages/alembic/config.py", line 601, in run_cmd
    fn(
  File "venv/lib/python3.12/site-packages/alembic/command.py", line 556, in heads
    heads = script.get_revisions(script.get_heads())
                                 ^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.12/site-packages/alembic/script/base.py", line 407, in get_heads
    return list(self.revision_map.heads)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
                                           ^^^^^^^^^^^^^^
  File "venv/lib/python3.12/site-packages/alembic/script/revision.py", line 143, in heads
    self._revision_map
  File "venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
                                           ^^^^^^^^^^^^^^
  File "venv/lib/python3.12/site-packages/alembic/script/revision.py", line 197, in _revision_map
    for revision in self._generator():
  File "venv/lib/python3.12/site-packages/alembic/script/base.py", line 149, in _load_revisions
    script = Script._from_filename(self, dir_name, filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.12/site-packages/alembic/script/base.py", line 1035, in _from_filename
    module = util.load_python_file(dir_, filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "venv/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 109, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 994, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "migrations/versions/be59ab2abe34_update_foreign_key_constraints.py", line 8, in <module>
    @dataclass
     ^^^^^^^^^
  File ".pyenv/versions/3.12.0/lib/python3.12/dataclasses.py", line 1266, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File ".pyenv/versions/3.12.0/lib/python3.12/dataclasses.py", line 1256, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.12.0/lib/python3.12/dataclasses.py", line 983, in _process_class
    and _is_type(type, cls, dataclasses, dataclasses.KW_ONLY,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".pyenv/versions/3.12.0/lib/python3.12/dataclasses.py", line 749, in _is_type
    ns = sys.modules.get(cls.__module__).__dict__
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute '__dict__'. Did you mean: '__dir__'?```

Versions.

• OS: Linux
• Python: 3.12.0
• Alembic: 1.13.1
• SQLAlchemy: 1.4.51
• Database: n/a
• DBAPI: n/a

Additional context Found also other project suffering from this issue: https://github.com/mkdocs/mkdocs/issues/3141

Have a nice day!

[CaselIT]

Hi,

The linked issues seems to provide the solution, ie manually adding the module to the sys.modules dict. I'm not sure if there is a reason for not adding them in alembic, @zzzeek do you know if it was left out on purpose?

[CaselIT]

Interestingly when alembic still supported py27 this code actively removed the module from sys.modules in the py2 code: https://github.com/sqlalchemy/alembic/blob/b1e5a23c0f53482dd06b013ce8a9453dd0e5eb5d/alembic/util/compat.py#L176-L241

But I guess if the current behaviour is wanted we could add the module to sys.modules, exec it, remove the module from sys.modules

[zzzeek]

Hi,

The linked issues seems to provide the solution, ie manually adding the module to the sys.modules dict. I'm not sure if there is a reason for not adding them in alembic, @zzzeek do you know if it was left out on purpose?

it is on purpose, since these aren't "modules" that are imported by "import", and I didnt think we wanted these python modules to live in the Python interpreter permanently until the interpreter ends..there could be hundreds of files taking up memory etc.

we do the same thing in mako FYI. so...might want to try over there too.

[CaselIT]

(which...how do you clean up ...shrugs)

well in python2 alembic was quite blunt about it del sys.modules[module_name]. Maybe it works also on python3?

Alternatively, since this seems a very rare case we do a "try current version; catch Attribute error and re-try with the module added in sys.modules"?

[zzzeek]

trying to clean up after the fact seems kind of messy

at the moment I feel like new config flag in alembic.ini that turns on "add to sys.modules ; note: migration files will remain persistent in the python interpreter until interpreter close unless removed manually" somethign like that

[CaselIT]

don't really like having yet another config knob, but it's probably the best option

[zzzeek]

any chance someone reported this as a shortcoming in python dataclasses?

[CaselIT]

I think what python does here is very similar to what sqlalchemy does to evaluate a string annotation: exec it using the module dict to use the proper globals.

Not sure if anything different could be done in the dataclass module

[CaselIT]

honestly I'm not 100% it makes sense adding a new config flag just for this, since it's utility seems very low.

I'm for closing with "won't fix", since the workaround is simple: define the dataclass on another module and import it from the migration.

[zzzeek]

OK that's fine

[kasium]

Thanks for having a look