Feature Request: Post-Study Trial Replay Mode

[A-Ivan]

Hi @AdamBebko ,

I have a feature request, I didn't find any information on this in the documentation, but I may have missed it.

It would be great to be able to have a post-study replay mode where the researcher could replay, in the Unity editor, a selected trial, based on the recorded data, to be able to perform a detailed inspection of the participant's behavior during the trial, from both a first-person and a third-person (bird's-eye/overview) perspective. This would also allow the researcher to inspect the data acquired for validity and, if needed, to run any additional algorithms to process additional information (i.e., image processing/segmentation of where the participant looked), which can be processing heavy and, therefore, shouldn't be run during run-time as it could impact the framerate and, therefore, the quality of the experiment.

This could also be useful for cued-recall interview as the researcher could show the first-person perspective to the participant and ask questions about it. This would provide more possibilities than simply video recording the VR screen during the study, which could also impact framerate during the study and would be limited to studies running on a computer. Video recording of Unity, from either of these perspectives, could be created by extending this replay mode with the Unity's Recorder feature.

An example of this replay mode feature can be seen in the EVE Framework - https://github.com/cog-ethz/EVE

Thank you. Ivan

[AdamBebko]

Funny you should suggest that. I'm working on that right now for one of our own projects. It's very difficult to get a smooth user experience though, so it might take a while to complete. It needs to pull data from both the main CSV output and also from any continuous trackers. And from multiple runs, so connecting all those pieces is proving to be a challenge in a user-friendly way. It also needs to be somewhat robust to changes in the experiment that happen between recording and playback.

[A-Ivan]

What a coincidence. Yeah, I imagine it might be difficult to import these CSV's back into Unity in a user-friendly way. The good thing is that since you have full control of how and where things are saved it makes some parts of the import easier (at least in cases where the experiment was run on a computer). Are you running this through the Unity editor during Edit or Play mode? I could see it be beneficial for both circumstances and pros and cons for both in terms of user-friendliness.

Another feature that may be interesting, which could go with this, is to draw paths (for instance using the line renderer) for the continuously tracked objects. This can be helpful for generating images from an overview perspective of the scene, which could be useful both to understand participant behavior in a visual way and for generating a figure, which could be used to show and explain what occurred in one of the trials.

[A-Ivan]

I wonder if this can be implemented with the Unity Timeline, where scrubbing through the timeline would change which data from the CSV file is being read.

For selecting the csv files, if the default folders are used for them, then maybe the user can just choose from a list of participant numbers and then trial numbers (I believe it will be tricky here to make the trials name presented meaningful),

[AdamBebko]

Ok, so I have a pretty robust solution for our purposes. I will be including it in the next release (v3), hopefully with some basic documentation for it. At the moment I wrote it mainly to record transforms, bit it's based on a generic script so it's fairly straightforward to extend this to record/playback custom components.

I need to clean it up a bit still and test it out outside of our own projects first.

[A-Ivan]

Hi @AdamBebko

This is great news, thank you for all of the information on all of the posts. I think the basic of recording and playing back of transforms, as long as the user can set which game object transforms to record this should be enough. I believe you just need to make sure that during playback that physics is disabled, just to ensure that physics won't cause problems (collision, gravity, etc.).

It seems like there will be many great improvements to the next release (v3), when do you expect to have it available?

[AdamBebko]

I’m aiming for before the holidays.. we’ll see if life will cooperate

On Dec 1, 2021, at 1:20 PM, Ivan Aguilar @.***> wrote:

Hi @AdamBebko https://github.com/AdamBebko This is great news, thank you for all of the information on all of the posts. I think the basic of recording and playing back transforms, as long as the user can set which game object transforms to record this should be enough (while disabling physics during playback to make sure the physics won't cause problems).

It seems like there will be many great improvements to the next release (v3), when do you expect to have it available?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BioMotionLab/TUX/issues/35#issuecomment-983933482, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKKFU435H7RIBLOE4OT2H7DUOZRODANCNFSM5FARUF3A. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[AdamBebko]

This is now complete, just need to write it up and add some documentation to make it a bit more useable. Will be in next version (coming soon hopefully)

[A-Ivan]

Sounds great, can't wait to see it. I can help out if needed.

[AdamBebko]

pushed to dev branch. Still needs documentation