[Bug]: not bound to a Session; lazy load operation of attribute 'action_event' cannot proceed

[abrichr]

Describe the bug

  File "/Users/abrichr/oa/OpenAdapt/openadapt/replay.py", line 84, in replay
    strategy = strategy_class(recording, **kwargs)
               |              |            -> {'instructions': 'complete the timesheet'}
               |              -> Recording(id=76, timestamp=1719443335.466385, monitor_width=1512, monitor_height=982, double_click_interval_seconds=0.5, doub...
               -> <class 'openadapt.strategies.visual.VisualReplayStrategy'>

  File "/Users/abrichr/oa/OpenAdapt/openadapt/strategies/visual.py", line 182, in __init__
    add_active_segment_descriptions(recording.processed_action_events)
    |                               |         -> <property object at 0x16c362f20>
    |                               -> Recording(id=76, timestamp=1719443335.466385, monitor_width=1512, monitor_height=982, double_click_interval_seconds=0.5, doub...
    -> <function add_active_segment_descriptions at 0x2b2745a20>

  File "/Users/abrichr/oa/OpenAdapt/openadapt/strategies/visual.py", line 109, in add_active_segment_descriptions
    window_segmentation = get_window_segmentation(action)
                          |                       -> ActionEvent(name='singleclick', timestamp=1719443344.59867, recording_timestamp=1719443335.466385, screenshot_timestamp=17194...
                          -> <function get_window_segmentation at 0x2cecfd090>

  File "/Users/abrichr/oa/OpenAdapt/openadapt/strategies/visual.py", line 394, in get_window_segmentation
    original_image = screenshot.cropped_image
                     |          -> <property object at 0x16c3be160>
                     -> Screenshot(id=4666, recording_timestamp=1719443335.466385, recording_id=76, timestamp=1719443349.534288)

  File "/Users/abrichr/oa/OpenAdapt/openadapt/models.py", line 762, in cropped_image
    action_event = self.action_event[-1]
                   |    -> <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x16c3bee80>
                   -> Screenshot(id=4666, recording_timestamp=1719443335.466385, recording_id=76, timestamp=1719443349.534288)

  File "/Users/abrichr/Library/Caches/pypoetry/virtualenvs/openadapt-VBXg4jpm-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 482, in __get__
    return self.impl.get(state, dict_)
           |    |    |   |      -> {'_sa_instance_state': <sqlalchemy.orm.state.InstanceState object at 0x2cf27e440>, 'id': 4666, 'timestamp': 1719443349.534288...
           |    |    |   -> <sqlalchemy.orm.state.InstanceState object at 0x2cf27e440>
           |    |    -> <function AttributeImpl.get at 0x115df4310>
           |    -> <sqlalchemy.orm.attributes.CollectionAttributeImpl object at 0x2929eba00>
           -> <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x16c3bee80>
  File "/Users/abrichr/Library/Caches/pypoetry/virtualenvs/openadapt-VBXg4jpm-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 942, in get
    value = self._fire_loader_callables(state, key, passive)
            |    |                      |      |    -> symbol('PASSIVE_OFF')
            |    |                      |      -> 'action_event'
            |    |                      -> <sqlalchemy.orm.state.InstanceState object at 0x2cf27e440>
            |    -> <function AttributeImpl._fire_loader_callables at 0x115df43a0>
            -> <sqlalchemy.orm.attributes.CollectionAttributeImpl object at 0x2929eba00>
  File "/Users/abrichr/Library/Caches/pypoetry/virtualenvs/openadapt-VBXg4jpm-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/attributes.py", line 978, in _fire_loader_callables
    return self.callable_(state, passive)
           |    |         |      -> symbol('PASSIVE_OFF')
           |    |         -> <sqlalchemy.orm.state.InstanceState object at 0x2cf27e440>
           |    -> <member 'callable_' of 'AttributeImpl' objects>
           -> <sqlalchemy.orm.attributes.CollectionAttributeImpl object at 0x2929eba00>
  File "/Users/abrichr/Library/Caches/pypoetry/virtualenvs/openadapt-VBXg4jpm-py3.10/lib/python3.10/site-packages/sqlalchemy/orm/strategies.py", line 863, in _load_for_state
    raise orm_exc.DetachedInstanceError(
          |       -> <class 'sqlalchemy.orm.exc.DetachedInstanceError'>
          -> <module 'sqlalchemy.orm.exc' from '/Users/abrichr/Library/Caches/pypoetry/virtualenvs/openadapt-VBXg4jpm-py3.10/lib/python3.1...

sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <Screenshot at 0x2cf27e470> is not bound to a Session; lazy load operation of attribute 'action_event' cannot proceed (Background on this error at: https://sqlalche.me/e/14/bhk3)

2024-06-26 19:11:22.065 | INFO     | openadapt.strategies.visual:__del__:258 - action_history=
[]

To Reproduce

python -m openadapt.replay VisualReplayStrategy --instructions "complete the timesheet" --capture

[KIRA009]

@abrichr This looks like its because the session you are fetching recording.processed_action_events from is closed, which means only direct relationships would work. action_event.screenshot would work, but any relationships on screenshot would fail, which is what is happening when cropped_image is called. The solution could be that in get_window_segmentation, you could create a read-only session that we can then attach the screenshot object to (session.add should work). Let me know if that fixes it

[abrichr]

@KIRA009 this should really be abstracted away somehow. Any idea when this regression was introduced? This code hasn't changed in a while.

Edit: it appears to have spontaneously resolved. The timesheet task has different timing than the calculator task (i.e. more segments take longer to describe). Could it be a race condition?

[KIRA009]

Doesn't really look like a race condition, especially because it looks like its running on the same process.

This code hasn't changed in a while.

Do you mean this was working after the changes we made to the different types of db sessions?

[abrichr]

ChatGPT:

---

The DetachedInstanceError in SQLAlchemy occurs when you try to access a related object or attribute that requires a database query, but the parent object is not attached to an active session. Here are steps to address this issue:

1. Ensure Session is Open

Make sure the session is still open when accessing the attribute.

2. Explicitly Load the Attribute

Load the required attributes while the session is active.

3. Attach to a Session

Reattach the detached instance to an active session.

Example Fixes

Fix 1: Eager Loading

Use joinedload to eagerly load the attribute when querying the parent instance.

from sqlalchemy.orm import joinedload

session = Session()
screenshot = session.query(Screenshot).options(joinedload(Screenshot.action_event)).filter_by(id=some_id).first()

Fix 2: Ensure Session is Open

Ensure the session is still open when accessing the lazy-loaded attribute.

session = Session()
screenshot = session.query(Screenshot).filter_by(id=some_id).first()

# Access the action_event attribute while the session is still open
action_event = screenshot.action_event

Fix 3: Reattach to a Session

Reattach the detached instance to an active session before accessing the attribute.

session = Session()
screenshot = session.query(Screenshot).filter_by(id=some_id).first()

# Detach and later reattach the instance
session.expunge(screenshot)

# Reattach to a session
session.add(screenshot)
action_event = screenshot.action_event

Choose the method that best fits your use case.

---

I think we want to use joinedload in the relevant functions in crud.py.

[abrichr]

This is happening because the session is closed at the end of https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/models.py#L103