New sqlalchemy warnings with aiida-core 2.5

[danielhollas]

Since we started testing AWB with aiida-core 2.5, we now have a ton of warnings coming from sqlalchemy

sqlalchemy.exc.SAWarning: Object of type <DbNode> not in session, 
add operation along 'DbUser.dbnodes' will not proceed
(This warning originated from the Session 'autoflush' process,
which was invoked automatically in response to a user-initiated operation.)

It's not clear where this is coming from.

Full stack trace

```python ________________________ test_structure_browser_widget _________________________ structure_data_object = @pytest.mark.usefixtures("aiida_profile_clean") def test_structure_browser_widget(structure_data_object): """Test the `StructureBrowserWidget`.""" > structure_browser_widget = awb.StructureBrowserWidget() tests/test_structures.py:88: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/aiidalab_widgets_base/structures.py:533: in __init__ options=sorted({"All"}.union({i[0] for i in qbuilder.iterall() if i[0]})), /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/aiidalab_widgets_base/structures.py:533: in options=sorted({"All"}.union({i[0] for i in qbuilder.iterall() if i[0]})), /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/aiida/orm/querybuilder.py:1042: in iterall for item in self._impl.iterall(self.as_dict(), batch_size): /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/aiida/storage/psql_dos/orm/querybuilder/main.py:166: in iterall with nullcontext() if session.in_nested_transaction() else self._backend.transaction(): # type: ignore[attr-defined] /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/contextlib.py:135: in __enter__ return next(self.gen) /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/aiida/storage/psql_dos/backend.py:266: in transaction with session.begin_nested() as savepoint: /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/session.py:1958: in begin_nested return self.begin(nested=True) /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/session.py:1924: in begin trans = trans._begin(nested=nested) :2: in _begin ??? /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py:139: in _go ret_value = fn(self, *arg, **kw) /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/session.py:1045: in _begin return SessionTransaction( /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/session.py:960: in __init__ self._take_snapshot() /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/session.py:1089: in _take_snapshot self.session.flush() /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/session.py:4341: in flush self._flush(objects) /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/session.py:4476: in _flush with util.safe_reraise(): /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py:146: in __exit__ raise exc_value.with_traceback(exc_tb) /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/session.py:4437: in _flush flush_context.execute() /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py:442: in execute postsort_actions = self._generate_actions() /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py:400: in _generate_actions if action.execute(self): /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py:535: in execute self.dependency_processor.presort_saves(uow, save_states) /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/dependency.py:494: in presort_saves uowcommit.register_object( /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py:320: in register_object util.warn( /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py:1815: in warn _warnings_warn(msg, exc.SAWarning) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ message = "Object of type not in session, add operation along 'DbUser.dbnodes' will not proceed" category = , stacklevel = 16 def _warnings_warn( message: Union[str, Warning], category: Optional[Type[Warning]] = None, stacklevel: int = 2, ) -> None: # adjust the given stacklevel to be outside of SQLAlchemy try: frame = sys._getframe(stacklevel) except ValueError: # being called from less than 3 (or given) stacklevels, weird, # but don't crash stacklevel = 0 except: # _getframe() doesn't work, weird interpreter issue, weird, # ok, but don't crash stacklevel = 0 else: stacklevel_found = warning_tag_found = False while frame is not None: # using __name__ here requires that we have __name__ in the # __globals__ of the decorated string functions we make also. # we generate this using {"__name__": fn.__module__} if not stacklevel_found and not re.match( _not_sa_pattern, frame.f_globals.get("__name__", "") ): # stop incrementing stack level if an out-of-SQLA line # were found. stacklevel_found = True # however, for the warning tag thing, we have to keep # scanning up the whole traceback if frame.f_code in _warning_tags: warning_tag_found = True (_suffix, _category) = _warning_tags[frame.f_code] category = category or _category message = f"{message} ({_suffix})" frame = frame.f_back # type: ignore[assignment] if not stacklevel_found: stacklevel += 1 elif stacklevel_found and warning_tag_found: break if category is not None: > warnings.warn(message, category, stacklevel=stacklevel + 1) E sqlalchemy.exc.SAWarning: Object of type not in session, add operation along 'DbUser.dbnodes' will not proceed /opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py:1890: SAWarning ```

[danielhollas]

@sphuber any idea what this is and why we are only seeing this with 2.5 and not 2.1?

[sphuber]

https://github.com/aiidateam/aiida-core/pull/6192

TLDR: warning is correctly emitted by sqlalchemy, but how AiiDA is built it is not of any real consequence. A real solution would have been too complicated so I worked around it to just silence it. See PR and commit message for details

[danielhollas]

Great, that's a relief, thank you for the info. However, I am confused, based on the linked PR I would think that we shouldn't see these warnings here since they should be filtered out?

[sphuber]

You are right to be confused, so was I 😅 The problem is that you are seeing this when running pytest. This is because pytest messes with the warning configuration and basically undoes the filtering... You have to manually filter it again in the pytest ini options, see: https://github.com/aiidateam/aiida-core/blob/589a3b2c03d44cebd26e88243ca34fcdb0e23ff4/pyproject.toml#L365

I couldn't find a way to have this always respected, so unfortunately all plugins using pytest for testing have to manually add this ignore rule

[danielhollas]

Aha, good to know, thank you for explaining. :+1: