Labelling Theory of Mind features from Movies

[jsmentch]

Background

As part of our research, we are interested in how we can use movies to learn more about the brain. One aspect of this is accurately labeling 'features' (like emotions, objects, sounds, etc) from movie clips that people have watched while their brains are scanned (e.g., fMRI). This is where we could use some help!

As a first step it would be really helpful to get annotations related to Theory of Mind from 2 short movie clips. What is Theory of Mind (ToM)? In short, you can think of it as one's ability to understand other people by attributing mental states (beliefs, desires, emotions) to them. ToM is very important for social interactions as it provides the ability to predict and interpret other people's behavior. For more background, search for 'Theory of Mind' on YouTube.

The features we want to get:

To identify regions of the brain related to ToM from fMRI movie data, we will need to label some features as seen in this paper. These include 4 conditions:

• “Social,” in which characters interact without engaging mental/emotional representations (e.g. cloud wrapping and handing over babies to storks, cloud and stork playing)
• “Pain,” in which a character is undergoing a physically painful event (e.g. bitten by a crocodile, electrocuted by an electric eel
• “Mental,” in which the viewer is led to think about the character's thoughts (e.g. a character who has just experienced pain watches others interacting happily, a character falsely believing he has been abandoned by his companion)
• “Control,” in which there are no specific character related events (e.g. flying birds, wide shot of clouds)

How to Help!

1. Download the short movie file found here
2. Go to the Feature Explorer website (Chrome/Brave browser is best, Safari/mobile not supported)
3. press Choose File and select your locally saved movie file.
4. On the next screen, press Add new empty feature. This adds a new feature with zeros for all time points for the length of the movie.
5. Watch the movie by pressing play or the spacebar
6. Edit the feature:
   • Toggle editing mode by pressing the red Editing button or 'e' on your keyboard.
   • With edit mode on, click within the 'Fine Timeline' box to label points related to the feature you are annotating.
   • If you miss-click, hold 'shift' and click to set points back to zero that are not related to ToM.
   • More playback options are described at the bottom of the site page (volume, speed, navigation, etc)
   • *Warning: if you navigate away from the page while working, you will lose your progress!
7. When you are done, save your file
   • Under 'Feature Export', edit the text box to say something like: "TP_social_jsmentch.tsv"
   • Press export Feature. This will let you save your annotated feature to your computer.
8. You should do all 4 features sequentially, adding more as you go. We want these features to not overlap with one another. Since you are starting a new feature each time, it will be called 'new' in the feature visualizer. Keep track of which is which by color, or, after you annotate a feature, restart the feature visualizer page and upload your completed, exported .tsv file. The feature will then have the same name as the filename and easier to keep track of.
9. Upload your saved feature files using this url

Once you have done this first movie, you can also give this one a shot.

Thank you for your interest and help!!!

Post any questions/comments on this issue and I will be happy to help.

[sheilajmenon]

Here is my saved feature file of the edited short film: https://drive.google.com/file/d/1ZXqkscDIkH5dPUC1_-nuazN9Ttx_nxzR/view?usp=sharing I apologize for it being so late. I also made a document with notes/specific moments in the film related to ToM- feel free to take a look: https://docs.google.com/document/d/17yOat8Fl9qGM0fTaLII6ep9XlAc5zdkKoNONu5f4GYE/edit?usp=sharing

I hope this data/set of annotations can be of some help with your research: this project is fascinating, and I’m interested to see how you can use Theory of Mind and the annotations in films to match with patterns in fMRI scans- to eventually predict people’s behavior and diagnose certain illnesses or mental disorders.

Thank you so much for the opportunity to add to this project: I was able to learn a lot about Theory of Mind! :)

[jsmentch]

@sheilajmenon thanks so much! This is a big help. If you'd like to do another movie, I've updated the above directions to include a link to a second movie at the end along with new upload instructions for everyone.

For anyone else interested in helping, we will need multiple annotations of these same features from different people, so please follow the same steps above - thanks!

[c-gabrieli]

Hi! I've completed my annotations for the feature and submitted it, here's a link to my notes on the movie clip as well as some resources from Youtube to learn a little bit about ToM: https://docs.google.com/document/d/1OCAHSarXU3GyN2W4lV3LHV8VMtuyLBb-CdV3YKFYBU8/edit?usp=sharing I'll include the links to the resources below here too: Explaining Theory of Mind: https://www.youtube.com/watch?v=xov5mF1NauU https://www.youtube.com/watch?v=x1oJA7dRi_k Theory of Mind Tests: https://www.youtube.com/watch?v=YGSj2zY2OEM https://www.youtube.com/watch?v=41jSdOQQpv0 I'll start working on annotating the second feature and update the document linked above with notes from the second movie clip. I've really enjoyed learning about ToM and am excited to continue working on this project!

[jsmentch]

@c-gabrieli excellent, thank you! I received the file and will keep an eye out for the next one. Stay tuned for more

[jsmentch]

@sheilajmenon, @c-gabrieli, Haleh, +anyone else interested in helping (@dinapaul?)

Upon further reading and looking at the ToM features we have, I think we should shift to annotating Social, Pain, and Mental features. I've updated the instructions above to explain how to get these features. It would be a great help if we could get a few different versions of these features from different people for The Present and Despicable Me.

More bonus background and reasoning:

- I've noticed that the 3 annotated ToM features we have so far are all very different and ToM is actually a difficult/complicated feature to annotate. This will make it hard to use as a feature in an fMRI analysis to clearly identify regions of the brain related to ToM. Typically in fMRI research, you use what is called a contrast to compare different tasks/stimuli/conditions (Like looking at faces vs looking at houses, which would locate the brain regions that process faces, for example in this classic [paper](https://www.jneurosci.org/content/17/11/4302)). - These original ToM features could still be useful if we do something like a reverse correlation analysis (look at parts of the brain that we know beforehand are related to ToM and see which parts of the movie most activate those regions. Then look up what was going on at those parts, was it ToM related? like in this [paper](http://saxelab.mit.edu/sites/default/files/publications/Richardson_Lisandrelli_Riobueno-Naylor_Saxe_2018.pdf), figure 4) - I linked it above in the instructions, but check out this [paper](http://saxelab.mit.edu/sites/default/files/publications/JacobyBruneauKosterHaleSaxe2015.pdf) to see how we will be able to hopefully localize the ToM network using these new features.

[sooyounga]

Hi, I have submitted my files for both movies. It was indeed fun to learn about ToM, understand more through the papers and links provided, and try to make clean categorization of different features to the best of my understanding and perception!

Just a couple of comments for the team/people helping: 1) The second movie when downloaded (as well as uploaded to the feature explorer) seems not to have its audio anymore. When directly played via the link, it did have the audio albeit slightly delayed. I ended up playing it using the direct link, recording the timestamps separately for each ToM feature, and then trying to match it on the feature explorer to generate the .tsv files. 2) The feature explorer hangs when you try to record the feature till the last time point (i.e. dragging your clicks on the 'Fine Timeline' box until the very end) and you won't be able to interact with the feature explorer anymore. I tried to label as close to the end as possible, but my last few time points might be missing data.

I look forward to continuing to learn about this and contributing more! Thank you so much :)

[jsmentch]

@sooyounga Great! thank you and thanks for the feedback as well

1. I was able to re-convert that second despicable me video file and it should be working with both audio and non-laggy video now. edited original post to have this new link. If anyone runs into issues with playing the file, lmk.
2. thanks for reporting, I will fix the bug. I opened an issue here https://github.com/jsmentch/feature_visualizer/issues/44 . If others encounter problems please post here or open an issue like that there in that repo - thanks!

edit: i fixed that bug now, but for people to use the fixed version, you will likely need to clear the cache/history of your browser for the feature visualizer page

[haleh2000m]

Hi! I have extracted my four features separately (Mental, Pain, Social, Control) for each movie.

Trying to extract ToM features with Feature Visualizer was an amazing experience. I learned a lot. Thank you!

[satra]

Thank you for all your efforts. We are going to close this issue and restart as relevant.