WIP: Add replay logging mechanism

[KIRA009]

What kind of change does this PR introduce? This PR addresses #757

Summary It adds logging to database during replay that will then be displayed on the dashboard. The database bloats very quickly because the logs include images, and python objects

Checklist

• [x] My code follows the style guidelines of OpenAdapt
• [x] I have performed a self-review of my code
• [ ] If applicable, I have added tests to prove my fix is functional/effective
• [x] I have linted my code locally prior to submission
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
• [x] New and existing unit tests pass locally with my changes

How can your code be run and tested?

Other information

[KIRA009]

@abrichr I have added the segmentation logging for visual replay strategy

[abrichr]

Excellent! Can you please share screenshots? 🙏

[KIRA009]

https://github.com/user-attachments/assets/b62adbb3-2a7d-4eaf-a7a2-b173acc1e098

@abrichr

[abrichr]

Thank you @KIRA009. Those images don't look right. If you set DEBUG = True at the top of visual.py and run the strategy again, you will see the images that are generated at each step. These are the images I would expect to see in the dashboard. Please let me know if I can help clarify further!

[KIRA009]

@abrichr I see images like these, the replay stops arbitrarily for me though, with the exception 2024-07-25 01:52:21.720 | WARNING | openadapt.strategies.visual:get_next_action_event:253 - exc=ValueError("'Settings gear icon' is not in list")

I am not sure if its because my recording is not great, or my prompt is not good, but this should log any segmentation returned by get_window_segmentation in visual.py

[abrichr]

exc=ValueError("'Settings gear icon' is not in list")

This is tracked here: https://github.com/OpenAdaptAI/OpenAdapt/issues/658

this should log any segmentation returned by get_window_segmentation in visual.py

Those images you pasted are indeed the image I was referring to in my previous comment. However, I don't see them in the video you shared.

[KIRA009]

@abrichr here's the video after merging the fix

https://github.com/user-attachments/assets/ca423ae1-7a98-4bcd-8b2e-b2c81b894c35

[abrichr]

@KIRA009 it looks like the screenshot is blank. Any idea why that is? Which application was open when the screenshot was taken?

[KIRA009]

@abrichr it was vs code I believe. You could try this branch on a recording you have and check the results