search

OpenAdaptAI / OpenAdapt

Open Source Generative Process Automation (i.e. Generative RPA). AI-First Process Automation with Large ([Language (LLMs) / Action (LAMs) / Multimodal (LMMs)] / Visual Language (VLMs)) Models
https://www.OpenAdapt.AI
MIT License
871 stars 115 forks source link

[Bug]: TypeErrors when replaying recording #352

Open flyguy712 opened 1 year ago

flyguy712 commented 1 year ago

Describe the bug

I expected the recording to fully playback, but what actually happened is that it only partially played back and then several 'TypeError' expressions appeared in the Terminal logs.

Note that my recording should have encompassed a workflow that extended beyond typing "flights.google.com" into the search bar and pressing "Enter".

Screen recording of the replay: https://github.com/OpenAdaptAI/OpenAdapt/assets/84868895/077443b3-447b-4d87-911c-ed8b687c7a31

Note that the logs below have been truncated to fit in the description for this issue...

Here are the logs from when I tried to replay the recording:

(openadapt-py3.10) taylorhorsager@Taylors-MacBook-Air OpenAdapt % python -m openadapt.replay NaiveReplayStrategy
2023-07-04 17:20:03.224 | INFO     | openadapt.config:<module>:146 - STOP_STRS=['oa.stop']
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - SPECIAL_CHAR_STOP_SEQUENCES=[['ctrl', 'ctrl', 'ctrl']]
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - STOP_SEQUENCES=[['o', 'a', '.', 's', 't', 'o', 'p'], ['ctrl', 'ctrl', 'ctrl']]
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - CACHE_DIR_PATH=.cache
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - CACHE_ENABLED=True
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - CACHE_VERBOSITY=0
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - DB_ECHO=False
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - DB_FNAME=openadapt.db
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - OPENAI_API_KEY=<set your api key in .env>
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - OPENAI_MODEL_NAME=gpt-3.5-turbo
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - RECORD_READ_ACTIVE_ELEMENT_STATE=False
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - REPLAY_STRIP_ELEMENT_STATE=True
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - IGNORE_WARNINGS=False
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - ACTION_TEXT_SEP=-
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - ACTION_TEXT_NAME_PREFIX=<
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - ACTION_TEXT_NAME_SUFFIX=>
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - SCRUB_ENABLED=True
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - SCRUB_CHAR=*
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - SCRUB_LANGUAGE=en
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - SCRUB_FILL_COLOR=255
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - SCRUB_CONFIG_TRF={'nlp_engine_name': 'spacy', 'models': [{'lang_code': 'en', 'model_name': 'en_core_web_trf'}]}
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - SCRUB_IGNORE_ENTITIES=['DATE_TIME', 'URL']
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - SCRUB_KEYS_HTML=['text', 'canonical_text', 'title', 'state', 'task_description', 'key_char', 'canonical_key_char', 'key_vk', 'children']
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - PLOT_PERFORMANCE=True
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - ROOT_DIRPATH=/Users/taylorhorsager/OpenAdapt
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - DB_FPATH=/Users/taylorhorsager/OpenAdapt/openadapt.db
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<
module>:146 - DB_URL=sqlite:////Users/taylorhorsager/OpenAdapt/openadapt.db
2023-07-04 17:20:03.225 | INFO     | openadapt.config:<module>:146 - DIRNAME_PERFORMANCE_PLOTS=performance
2023-07-04 17:20:03.817 | INFO     | __main__:replay:27 - strategy_name='NaiveReplayStrategy'
2023-07-04 17:20:05.460 | INFO     | __main__:replay:40 - strategy_class=<class 'openadapt.strategies.naive.NaiveReplayStrategy'>
2023-07-04 17:20:05.460 | INFO     | __main__:replay:43 - strategy=<openadapt.strategies.naive.NaiveReplayStrategy object at 0x12b530a30>
2023-07-04 17:20:11.297 | INFO     | openadapt.events:get_events:39 - num_process_iters=0 num_action_events=700 num_window_events=9 num_screenshots=238
2023-07-04 17:20:11.297 | INFO     | openadapt.events:process_events:600 - before num_action_events=700 num_window_events=9 num_screenshots=238 num_total=947
2023-07-04 17:20:11.308 | INFO     | openadapt.events:get_group_idx_tups:432 - len(to_merge)=38 group_idx_tups=[(0, 36), (36, 38)]
2023-07-04 17:20:11.396 | INFO     | openadapt.events:get_group_idx_tups:432 - len(to_merge)=6 group_idx_tups=[(0, 6)]
2023-07-04 17:20:11.432 | INFO     | openadapt.events:merge_consecutive_action_events:564 - name='keyboard' num_events_removed=41
2023-07-04 17:20:11.432 | INFO     | openadapt.events:discard_unused_events:586 - referred_timestamp_key='window_event_timestamp' num_referred_events_removed=0
2023-07-04 17:20:11.433 | INFO     | openadapt.events:discard_unused_events:586 - referred_timestamp_key='screenshot_timestamp' num_referred_events_removed=39
2023-07-04 17:20:11.523 | INFO     | openadapt.events:merge_consecutive_action_events:564 - name='mouse_move' num_events_removed=623
2023-07-04 17:20:11.523 | INFO     | openadapt.events:discard_unused_events:586 - referred_timestamp_key='window_event_timestamp' num_referred_events_removed=0
2023-07-04 17:20:11.523 | INFO     | openadapt.events:discard_unused_events:586 - referred_timestamp_key='screenshot_timestamp' num_referred_events_removed=163
2023-07-04 17:20:11.523 | INFO     | openadapt.events:merge_consecutive_action_events:564 - name='mouse_scroll' num_events_removed=0
2023-07-04 17:20:11.523 | INFO     | openadapt.events:discard_unused_events:586 - referred_timestamp_key='window_event_timestamp' num_referred_events_removed=0
2023-07-04 17:20:11.523 | INFO     | openadapt.events:discard_unused_events:586 - referred_timestamp_key='screenshot_timestamp' num_referred_events_removed=0
2023-07-04 17:20:11.524 | INFO     | openadapt.events:merge_consecutive_action_events:564 - name='redundant_mouse_move' num_events_removed=11
2023-07-04 17:20:11.524 | INFO     | openadapt.events:discard_unused_events:586 - referred_timestamp_key='window_event_timestamp' num_referred_events_removed=2
2023-07-04 17:20:11.524 | INFO     | openadapt.events:discard_unused_events:586 - referred_timestamp_key='screenshot_timestamp' num_referred_events_removed=11
2023-07-04 17:20:11.528 | INFO     | openadapt.events:get_timestamp_mappings:282 - double_click_distance=5
2023-07-04 17:20:11.528 | INFO     | openadapt.events:get_timestamp_mappings:288 - double_click_interval=0.5
2023-07-04 17:20:11.547 | INFO     | openadapt.events:get_timestamp_mappings:282 - double_click_distance=5
2023-07-04 17:20:11.547 | INFO     | openadapt.events:get_timestamp_mappings:288 - double_click_interval=0.5
2023-07-04 17:20:11.553 | INFO     | openadapt.events:get_timestamp_mappings:282 - double_click_distance=5
2023-07-04 17:20:11.553 | INFO     | openadapt.events:get_timestamp_mappings:288 - double_click_interval=0.5
2023-07-04 17:20:11.583 | INFO     | openadapt.events:merge_consecutive_action_events:564 - name='mouse_click' num_events_removed=11
2023-07-04 17:20:11.583 | INFO     | openadapt.events:discard_unused_events:586 - referred_timestamp_key='window_event_timestamp' num_referred_events_removed=0
2023-07-04 17:20:11.583 | INFO     | openadapt.events:discard_unused_events:586 - referred_timestamp_key='screenshot_timestamp' num_referred_events_removed=11
2023-07-04 17:20:11.583 | INFO     | openadapt.events:process_events:636 - after num_action_events_=14 num_window_events_=7 num_screenshots_=14 num_total=947
2023-07-04 17:20:11.583 | INFO     | openadapt.events:process_events:640 - pct_action_events=0.02 pct_window_events=0.7777777777777778 pct_screenshots=0.058823529411764705 pct_total=0.03695881731784583
2023-07-04 17:20:11.583 | INFO     | openadapt.events:get_events:83 - duration=5.993409633636475
2023-07-04 17:20:11.584 | INFO     | openadapt.strategies.naive:get_next_action_event:58 - self.action_event_idx=0 of num_action_events=14: action_event=ActionEvent(id=3282, name='singleclick', timestamp=1689829367.3202674, recording_timestamp=1688512445.664324, screenshot_timestamp=1689829369.403077, window_event_timestamp=1689829354.8723857, mouse_x=1360.8984375, mouse_y=221.7421875, mouse_button_name='left')
2023-07-04 17:20:11.600 | INFO     | openadapt.strategies.base:run:67 - action_event=

[TRUNCATED SECTION]

 'recording_timestamp': 1688512445.664324,
 'screenshot_timestamp': 1689829369.403077,
 'timestamp': 1689829367.3202674,
 'window_event_timestamp': 1689829354.8723857}
2023-07-04 17:20:12.197 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:20:12.197 | INFO     | openadapt.strategies.naive:get_next_action_event:58 - self.action_event_idx=1 of num_action_events=14: action_event=ActionEvent(id=3283, name='singleclick', timestamp=1689829367.4435203, recording_timestamp=1688512445.664324, screenshot_timestamp=1689829374.6669717, window_event_timestamp=1689829373.4887724, mouse_x=1158.29296875, mouse_y=254.28515625, mouse_button_name='left')
2023-07-04 17:20:12.201 | INFO     | openadapt.strategies.base:run:67 - action_event=

[TRUNCATED SECTION]

 'recording_timestamp': 1688512445.664324,
 'screenshot_timestamp': 1689829374.6669717,
 'timestamp': 1689829367.4435203,
 'window_event_timestamp': 1689829373.4887724}
2023-07-04 17:20:12.678 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:20:12.679 | INFO     | openadapt.strategies.naive:get_next_action_event:58 - self.action_event_idx=2 of num_action_events=14: action_event=ActionEvent(id=3268, name='type', timestamp=1689829369.2004812, recording_timestamp=1688512445.664324, screenshot_timestamp=1689829376.5902064, window_event_timestamp=1689829373.4887724, text='f-l-i-g-h-t-s-.-g-o-o-g-l-e-.-c-o-m', canonical_text='f-l-i-g-h-t-s-.-g-o-o-g-l-e-.-c-o-m')
2023-07-04 17:20:12.681 | INFO     | openadapt.strategies.base:log_fps:88 - fps=1.83
2023-07-04 17:20:12.684 | INFO     | openadapt.strategies.base:run:67 - action_event=
{'canonical_text': 'f-l-i-g-h-t-s-.-g-o-o-g-l-e-.-c-o-m',
 'children': [{'canonical_key_char': 'f',
               'id': 2703,
               'key_char': 'f',
               'key_vk': '3',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829376.5902064,
               'timestamp': 1689829376.6233482,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 'f',
               'id': 2704,
               'key_char': 'f',
               'key_vk': '3',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829376.6639483,
               'timestamp': 1689829376.7030604,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 'l',
               'id': 2705,
               'key_char': 'l',
               'key_vk': '37',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829377.3813183,
               'timestamp': 1689829377.3993464,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 'l',
               'id': 2706,
               'key_char': 'l',
               'key_vk': '37',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829377.460672,
               'timestamp': 1689829377.4658744,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 'i',
               'id': 2707,
               'key_char': 'i',
               'key_vk': '34',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829377.814631,
               'timestamp': 1689829377.8271708,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 'i',
               'id': 2708,
               'key_char': 'i',
               'key_vk': '34',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829377.9361246,
               'timestamp': 1689829377.9385128,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 'g',
               'id': 2709,
               'key_char': 'g',
               'key_vk': '5',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829378.1630144,
               'timestamp': 1689829378.1677318,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 'g',
               'id': 2710,
               'key_char': 'g',
               'key_vk': '5',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829378.2362535,
               'timestamp': 1689829378.254168,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 'h',
               'id': 2711,
               'key_char': 'h',
               'key_vk': '4',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829378.2362535,
               'timestamp': 1689829378.2817876,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 't',
               'id': 2712,
               'key_char': 't',
               'key_vk': '17',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829378.370367,
               'timestamp': 1689829378.3898249,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 'h',
               'id': 2713,
               'key_char': 'h',
               'key_vk': '4',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829378.370367,
               'timestamp': 1689829378.3904893,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 't',
               'id': 2714,
               'key_char': 't',
               'key_vk': '17',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829378.4493673,
               'timestamp': 1689829378.482988,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 's',
               'id': 2715,
               'key_char': 's',
               'key_vk': '1',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829378.5164297,
               'timestamp': 1689829378.5516894,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 's',
               'id': 2716,
               'key_char': 's',
               'key_vk': '1',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829378.6257262,
               'timestamp': 1689829378.639836,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': '.',
               'id': 2717,
               'key_char': '.',
               'key_vk': '47',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829379.2210717,
               'timestamp': 1689829379.2436266,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': '.',
               'id': 2718,
               'key_char': '.',
               'key_vk': '47',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829379.3192923,
               'timestamp': 1689829379.363737,
               'window_event_timestamp': 1689829373.4887724},
              {'canonical_key_char': 'g',
               'id': 2719,
               'key_char': 'g',
               'key_vk': '5',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829380.0573847,
               'timestamp': 1689829380.0770197,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'g',
               'id': 2720,
               'key_char': 'g',
               'key_vk': '5',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829380.1340878,
               'timestamp': 1689829380.1665602,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'o',
               'id': 2721,
               'key_char': 'o',
               'key_vk': '31',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829380.394582,
               'timestamp': 1689829380.414233,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'o',
               'id': 2722,
               'key_char': 'o',
               'key_vk': '31',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829380.4677682,
               'timestamp': 1689829380.4823544,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'o',
               'id': 2723,
               'key_char': 'o',
               'key_vk': '31',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829380.556412,
               'timestamp': 1689829380.5714889,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'o',
               'id': 2724,
               'key_char': 'o',
               'key_vk': '31',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829380.7045507,
               'timestamp': 1689829380.737289,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'g',
               'id': 2725,
               'key_char': 'g',
               'key_vk': '5',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829380.7391899,
               'timestamp': 1689829380.751954,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'g',
               'id': 2726,
               'key_char': 'g',
               'key_vk': '5',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829380.826117,
               'timestamp': 1689829380.841954,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'l',
               'id': 2727,
               'key_char': 'l',
               'key_vk': '37',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829380.8965106,
               'timestamp': 1689829380.9361498,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'l',
               'id': 2728,
               'key_char': 'l',
               'key_vk': '37',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829381.0113325,
               'timestamp': 1689829381.0221906,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'e',
               'id': 2729,
               'key_char': 'e',
               'key_vk': '14',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829381.0113325,
               'timestamp': 1689829381.0451894,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'e',
               'id': 2730,
               'key_char': 'e',
               'key_vk': '14',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829381.0933523,
               'timestamp': 1689829381.112281,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': '.',
               'id': 2731,
               'key_char': '.',
               'key_vk': '47',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829381.1699114,
               'timestamp': 1689829381.2019842,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': '.',
               'id': 2732,
               'key_char': '.',
               'key_vk': '47',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829381.2844787,
               'timestamp': 1689829381.3204713,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'c',
               'id': 2733,
               'key_char': 'c',
               'key_vk': '8',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829381.4728346,
               'timestamp': 1689829381.4963942,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'c',
               'id': 2734,
               'key_char': 'c',
               'key_vk': '8',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829381.559251,
               'timestamp': 1689829381.630717,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'o',
               'id': 2735,
               'key_char': 'o',
               'key_vk': '31',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829381.6674414,
               'timestamp': 1689829381.6770205,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'o',
               'id': 2736,
               'key_char': 'o',
               'key_vk': '31',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829381.7536216,
               'timestamp': 1689829381.7965622,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'm',
               'id': 2737,
               'key_char': 'm',
               'key_vk': '46',
               'name': 'press',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829381.8346872,
               'timestamp': 1689829381.8534756,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_char': 'm',
               'id': 2738,
               'key_char': 'm',
               'key_vk': '46',
               'name': 'release',
               'parent_id': 3268,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829381.9144,
               'timestamp': 1689829381.9237363,
               'window_event_timestamp': 1689829379.9025235}],
 'id': 3268,
 'name': 'type',
 'recording_timestamp': 1688512445.664324,
 'screenshot_timestamp': 1689829376.5902064,
 'text': 'f-l-i-g-h-t-s-.-g-o-o-g-l-e-.-c-o-m',
 'timestamp': 1689829369.2004812,
 'window_event_timestamp': 1689829373.4887724}
2023-07-04 17:20:13.452 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:20:13.453 | INFO     | openadapt.strategies.naive:get_next_action_event:58 - self.action_event_idx=3 of num_action_events=14: action_event=ActionEvent(id=3269, name='type', timestamp=1689829369.5841374, recording_timestamp=1688512445.664324, screenshot_timestamp=1689829382.2931798, window_event_timestamp=1689829379.9025235, text='<enter>', canonical_text='<36>')
2023-07-04 17:20:13.453 | INFO     | openadapt.strategies.base:log_fps:88 - fps=1.61
2023-07-04 17:20:13.453 | INFO     | openadapt.strategies.base:run:67 - action_event=
{'canonical_text': '<36>',
 'children': [{'canonical_key_vk': '36',
               'id': 2739,
               'key_name': 'enter',
               'name': 'press',
               'parent_id': 3269,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829382.2931798,
               'timestamp': 1689829382.3073926,
               'window_event_timestamp': 1689829379.9025235},
              {'canonical_key_vk': '36',
               'id': 2740,
               'key_name': 'enter',
               'name': 'release',
               'parent_id': 3269,
               'recording_timestamp': 1688512445.664324,
               'screenshot_timestamp': 1689829382.3493192,
               'timestamp': 1689829382.4334404,
               'window_event_timestamp': 1689829379.9025235}],
 'id': 3269,
 'name': 'type',
 'recording_timestamp': 1688512445.664324,
 'screenshot_timestamp': 1689829382.2931798,
 'text': '<enter>',
 'timestamp': 1689829369.5841374,
 'window_event_timestamp': 1689829379.9025235}
2023-07-04 17:20:13.997 | WARNING  | openadapt.window:get_active_window_state:43 - exc=TypeError("'NoneType' object is not callable")
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/taylorhorsager/OpenAdapt/openadapt/replay.py", line 54, in <module>
    fire.Fire(replay)
  File "/Users/taylorhorsager/Library/Caches/pypoetry/virtualenvs/openadapt-IciE2gK5-py3.10/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/Users/taylorhorsager/Library/Caches/pypoetry/virtualenvs/openadapt-IciE2gK5-py3.10/lib/python3.10/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/Users/taylorhorsager/Library/Caches/pypoetry/virtualenvs/openadapt-IciE2gK5-py3.10/lib/python3.10/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/Users/taylorhorsager/OpenAdapt/openadapt/replay.py", line 45, in replay
    strategy.run()
  File "/Users/taylorhorsager/OpenAdapt/openadapt/strategies/base.py", line 47, in run
    window_event = models.WindowEvent.get_active_window_event()
  File "/Users/taylorhorsager/OpenAdapt/openadapt/models.py", line 305, in get_active_window_event
    return WindowEvent(**window.get_active_window_data())
TypeError: openadapt.models.WindowEvent() argument after ** must be a mapping, not NoneType

Here are the logs from making the recording:

(openadapt-py3.10) taylorhorsager@Taylors-MacBook-Air OpenAdapt % python -m openadapt.record "testing out openadapt"
2023-07-04 17:14:04.631 | INFO     | openadapt.config:<module>:146 - STOP_STRS=['oa.stop']
2023-07-04 17:14:04.631 | INFO     | openadapt.config:<module>:146 - SPECIAL_CHAR_STOP_SEQUENCES=[['ctrl', 'ctrl', 'ctrl']]
2023-07-04 17:14:04.631 | INFO     | openadapt.config:<module>:146 - STOP_SEQUENCES=[['o', 'a', '.', 's', 't', 'o', 'p'], ['ctrl', 'ctrl', 'ctrl']]
2023-07-04 17:14:04.631 | INFO     | openadapt.config:<module>:146 - CACHE_DIR_PATH=.cache
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - CACHE_ENABLED=True
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - CACHE_VERBOSITY=0
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - DB_ECHO=False
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - DB_FNAME=openadapt.db
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - OPENAI_API_KEY=<set your api key in .env>
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - OPENAI_MODEL_NAME=gpt-3.5-turbo
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - RECORD_READ_ACTIVE_ELEMENT_STATE=False
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - REPLAY_STRIP_ELEMENT_STATE=True
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - IGNORE_WARNINGS=False
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - ACTION_TEXT_SEP=-
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - ACTION_TEXT_NAME_PREFIX=<
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - ACTION_TEXT_NAME_SUFFIX=>
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - SCRUB_ENABLED=True
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - SCRUB_CHAR=*
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - SCRUB_LANGUAGE=en
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - SCRUB_FILL_COLOR=255
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - SCRUB_CONFIG_TRF={'nlp_engine_name': 'spacy', 'models': [{'lang_code': 'en', 'model_name': 'en_core_web_trf'}]}
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - SCRUB_IGNORE_ENTITIES=['DATE_TIME', 'URL']
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - SCRUB_KEYS_HTML=['text', 'canonical_text', 'title', 'state', 'task_description', 'key_char', 'canonical_key_char', 'key_vk', 'children']
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - PLOT_PERFORMANCE=True
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - ROOT_DIRPATH=/Users/taylorhorsager/OpenAdapt
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - DB_FPATH=/Users/taylorhorsager/OpenAdapt/openadapt.db
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - DB_URL=sqlite:////Users/taylorhorsager/OpenAdapt/openadapt.db
2023-07-04 17:14:04.632 | INFO     | openadapt.config:<module>:146 - DIRNAME_PERFORMANCE_PLOTS=performance
2023-07-04 17:14:05.659 | INFO     | __main__:wrapper_logging:91 -  -> Enter: record(testing out openadapt)
2023-07-04 17:14:05.663 | INFO     | __main__:record:657 - task_description='testing out openadapt'
2023-07-04 17:14:05.664 | INFO     | __main__:wrapper_logging:91 -  -> Enter: create_recording(testing out openadapt)
2023-07-04 17:14:05.710 | INFO     | __main__:create_recording:564 - recording=Recording(id=7, timestamp=1688512445.664324, monitor_width=2560, monitor_height=1440, double_click_interval_seconds=0.5, double_click_distance_pixels=5, platform='darwin', task_description='testing out openadapt')
2023-07-04 17:14:05.711 | INFO     | __main__:wrapper_logging:95 -  <- Leave: create_recording(Recording(id=7, timestamp=1688512445.664324, monitor_width=2560, monitor_height=1440, double_click_interval_seconds=0.5, double_click_distance_pixels=5, platform='darwin', task_description='testing out openadapt'))
2023-07-04 17:14:05.712 | INFO     | __main__:wrapper_logging:91 -  -> Enter: read_window_events(<queue.Queue object at 0x1463427d0>, <multiprocessing.synchronize.Event object at 0x1463aa4a0>, 1688512445.664324)
2023-07-04 17:14:05.718 | INFO     | __main__:read_window_events:432 - starting
2023-07-04 17:14:05.723 | INFO     | __main__:read_screen_events:405 - starting
2023-07-04 17:14:05.723 | INFO     | __main__:read_screen_events:405 - starting
2023-07-04 17:14:05.738 | INFO     | __main__:process_events:135 - starting
2023-07-04 17:14:05.982 | INFO     | __main__:read_window_events:451 - _window_data={'title': 'Terminal OpenAdapt — poetry shell ▸ Python — 80×77', 'left': 163, 'top': 79, 'width': 585, 'height': 1113, 'window_id': 65996}
2023-07-04 17:14:07.972 | INFO     | __mp_main__:wrapper_logging:91 -  -> Enter: write_events(window, <function write_window_event at 0x15e27b490>, <multiprocessing.queues.Queue object at 0x10310a4d0>, <multiprocessing.queues.Queue object at 0x15e262e30>, 1688512445.664324, <multiprocessing.synchronize.Event object at 0x15e263160>)
2023-07-04 17:14:07.977 | INFO     | __mp_main__:write_events:276 - event_type='window' starting
2023-07-04 17:14:07.997 | INFO     | __mp_main__:wrapper_logging:91 -  -> Enter: performance_stats_writer(<multiprocessing.queues.Queue object at 0x1035024d0>, 1688512445.664324, <multiprocessing.synchronize.Event object at 0x15de3ae30>)
2023-07-04 17:14:08.000 | INFO     | __mp_main__:wrapper_logging:91 -  -> Enter: write_events(screen, <function write_screen_event at 0x167c07400>, <multiprocessing.queues.Queue object at 0x104cc64d0>, <multiprocessing.queues.Queue object at 0x167beee30>, 1688512445.664324, <multiprocessing.synchronize.Event object at 0x167bef160>)
2023-07-04 17:14:08.003 | INFO     | __mp_main__:performance_stats_writer:482 - performance stats writer starting
2023-07-04 17:14:08.006 | INFO     | __mp_main__:write_events:276 - event_type='screen' starting
2023-07-04 17:14:08.022 | INFO     | __mp_main__:memory_writer:504 - Memory writer starting
2023-07-04 17:14:08.034 | INFO     | __mp_main__:wrapper_logging:91 -  -> Enter: write_events(action, <function write_action_event at 0x163927250>, <multiprocessing.queues.Queue object at 0x1009864d0>, <multiprocessing.queues.Queue object at 0x16390ee30>, 1688512445.664324, <multiprocessing.synchronize.Event object at 0x16390f160>)
2023-07-04 17:14:08.039 | INFO     | __mp_main__:write_events:276 - event_type='action' starting
2023-07-04 17:14:24.747 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:24.748 | INFO     | __main__:read_window_events:451 - _window_data={'title': 'Google Chrome New Tab', 'left': 937, 'top': 194, 'width': 1351, 'height': 794, 'window_id': 65998}
2023-07-04 17:14:26.715 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:29.868 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:33.127 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:33.130 | INFO     | __main__:read_window_events:451 - _window_data={'title': 'Google Chrome', 'left': 1055, 'top': 221, 'width': 1115, 'height': 408, 'window_id': 66001}
2023-07-04 17:14:36.521 | WARNING  | openadapt.window:get_active_window_state:43 - exc=TypeError("'NoneType' object is not callable")
2023-07-04 17:14:38.834 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:38.836 | INFO     | __main__:read_window_events:451 - _window_data={'title': 'Google Chrome Google Flights - Find Cheap Flight Options & Track Prices', 'left': 937, 'top': 194, 'width': 1351, 'height': 794, 'window_id': 65998}
2023-07-04 17:14:41.050 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:43.415 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:45.669 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:48.023 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:50.377 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:50.379 | INFO     | __main__:read_window_events:451 - _window_data={'title': 'Google Chrome Salt Lake City to Minneapolis | Google Flights', 'left': 937, 'top': 194, 'width': 1351, 'height': 794, 'window_id': 65998}
2023-07-04 17:14:52.802 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:55.086 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:57.610 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:14:59.919 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:15:02.108 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:15:04.450 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:15:04.452 | INFO     | __main__:read_window_events:451 - _window_data={'title': 'Google Chrome Minneapolis to Salt Lake City | Google Flights', 'left': 937, 'top': 194, 'width': 1351, 'height': 794, 'window_id': 65998}
2023-07-04 17:15:06.666 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:15:06.667 | INFO     | __main__:read_window_events:451 - _window_data={'title': 'Google Chrome Round trip to Minneapolis | Google Flights', 'left': 937, 'top': 194, 'width': 1351, 'height': 794, 'window_id': 65998}
^C2023-07-04 17:15:07.533 | INFO     | __mp_main__:write_events:286 - event_type='action' done
2023-07-04 17:15:07.534 | INFO     | __mp_main__:wrapper_logging:95 -  <- Leave: write_events(None)
2023-07-04 17:15:07.533 | INFO     | __mp_main__:write_events:286 - event_type='window' done
2023-07-04 17:15:07.534 | INFO     | __mp_main__:wrapper_logging:95 -  <- Leave: write_events(None)
2023-07-04 17:15:07.734 | INFO     | __main__:process_events:190 - done
2023-07-04 17:15:07.734 | INFO     | __main__:wrapper_logging:95 -  <- Leave: process_events(None)
2023-07-04 17:15:07.735 | INFO     | __main__:read_screen_events:412 - done
2023-07-04 17:15:08.147 | INFO     | __main__:log_memory_usage:65 - source='File "/Users/taylorhorsager/Library/Caches/pypoetry/virtualenvs/openadapt-IciE2gK5-py3.10/lib/python3.10/site-packages/mss/darwin.py", line 226'
2023-07-04 17:15:08.148 | INFO     | __main__:log_memory_usage:66 -     new_KiB=11289610.91015625 total_KiB=11289610.91015625 new_blocks=392 total_blocks=392
2023-07-04 17:15:08.149 | INFO     | __main__:log_memory_usage:65 - source='File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/reduction.py", line 51'
2023-07-04 17:15:08.149 | INFO     | __main__:log_memory_usage:66 -     new_KiB=64802.060546875 total_KiB=64802.6240234375 new_blocks=5 total_blocks=16
2023-07-04 17:15:08.150 | INFO     | __main__:log_memory_usage:65 - source='File "/Users/taylorhorsager/Library/Caches/pypoetry/virtualenvs/openadapt-IciE2gK5-py3.10/lib/python3.10/site-packages/atomacos/_converter.py", line 32'
2023-07-04 17:15:08.151 | INFO     | __main__:log_memory_usage:66 -     new_KiB=5833.7001953125 total_KiB=5833.7001953125 new_blocks=21307 total_blocks=21307
2023-07-04 17:15:17.168 | INFO     | __main__:log_memory_usage:69 - trace_str=
                                      types |   # objects |   total size
=========================================== | =========== | ============
                                  bytearray |         197 |     10.82 GB
                                _io.BytesIO |           3 |     63.28 MB
                                      tuple |      314180 |     21.51 MB
                                        str |       34321 |      8.94 MB
                                       list |       21078 |      4.45 MB
                        objc.pyobjc_unicode |       43558 |      3.84 MB
                                       dict |       10617 |      3.23 MB
  <objective-c class NSTaggedPointerString> |       19695 |      1.05 MB
           <objective-c class __NSCFString> |       19266 |      1.03 MB
                                      float |       41849 |    980.84 KB
   <objective-c class __NSCFConstantString> |        8287 |    453.20 KB
                 atomacos._converter.CGRect |        5718 |    402.05 KB
                atomacos._converter.CGPoint |        5156 |    281.97 KB
                                        int |        8838 |    242.36 KB
                 atomacos._converter.CGSize |        4227 |    231.16 KB
2023-07-04 17:15:17.169 | INFO     | __main__:record:782 - joining...
2023-07-04 17:15:17.627 | WARNING  | openadapt.window._macos:get_active_window_state:42 - exc=TypeError('Cannot pickle Objective-C objects')
2023-07-04 17:15:17.628 | INFO     | __main__:wrapper_logging:95 -  <- Leave: read_window_events(None)
2023-07-04 17:17:52.388 | INFO     | __mp_main__:write_events:286 - event_type='screen' done
2023-07-04 17:17:52.390 | INFO     | __mp_main__:wrapper_logging:95 -  <- Leave: write_events(None)
2023-07-04 17:17:52.502 | INFO     | __mp_main__:performance_stats_writer:496 - performance stats writer done
2023-07-04 17:17:52.503 | INFO     | __mp_main__:wrapper_logging:95 -  <- Leave: performance_stats_writer(None)
2023-07-04 17:17:52.506 | INFO     | __mp_main__:memory_writer:531 - Memory writer done
Deallocating objective-C class __NSCFConstantString2023-07-04 17:17:53.861 | INFO     | openadapt.utils:plot_performance:494 - fpath='performance/performance-1688512445.664324.png'
2023-07-04 17:17:54.381 | INFO     | __main__:record:798 - saved recording_timestamp=1688512445.664324
2023-07-04 17:17:54.383 | INFO     | __main__:wrapper_logging:95 -  <- Leave: record(None)
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 40 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

To Reproduce

I use macOS Ventura 13.1, Apple M1, 8 GB memory. The recording was created in a new/empty shell. Steps to reproduce:

  1. Open Terminal and input 'cd OpenAdapt' followed by 'poetry shell' followed by 'python -m openadapt.record "testing out openadapt"'
  2. Click on the search bar in Google Chrome (which should already be open on the desktop) and type 'flights.google.com' followed by 'Enter'
  3. Click on the destination for the flight and type 'MSP'
  4. Click the 'Search' button
  5. Click on the 'Airlines' dropdown
  6. Scroll down with the mouse wheel until you can see 'Delta' and select the 'Only' button (note that this button appears when the cursor is hovering over the right side of the row)
  7. Click the 'X' to close the dropdown
  8. Select the first departure flight that is displayed
  9. Select the first return flight that is displayed
  10. Select the Terminal and input 'CTRL+C' to stop the recording
  11. After the recording has finished processing, input 'python -m openadapt.replay NaiveReplayStrategy'
  12. BUG - the replay only displays part of the recording and the Terminal displays 'TypeError' codes
0dm commented 1 year ago

Thanks for the detailed report, I think it should be fixed in #323 - __NSCFData was not supported