Website to test working memory

fmichonneau commented 6 years ago

the site (http://cat.xula.edu/thinker/memory/working/serial) we use to test working memory requires Flash. As it's phasing out, it would be great to find a replacement that use another technology

ErinBecker commented 6 years ago

Thanks @fmichonneau. That's good to know. If anyone has any suggestions, please add to this issue.

kariljordan commented 5 years ago

Here is an interactive test from the Open Cognition Lab, but I can't tell if it requires flash.

fmichonneau commented 5 years ago

the one mentioned by Kari doesn't require Flash

ErinBecker commented 5 years ago

I taught with @symulation and @reedacartwright this week and we had some difficulties with the new memory game.

Not all of the symbols render on every operating system
This activity takes a LOT longer than the previous version. Probably 3 minutes instead of 30 seconds.
The result isn't a number of items that they could remember, so isn't directly comparable to the 7 plus or minus 2 rule. It was a score on some normalized scale.
The activity doesn't work with the chunking exercise. Much harder to make stories/connections between symbols than words and also not enough time to repeat the exercise to illustrate chunking.
There is an optional survey at the end before they get their results. You don't need to answer the questions, but there are about 5 pages of questions, so it's another few clicks. Also, the survey asks some awkward questions (seem to be randomized from a set, but the first time I took it there was a question about people from a particular location being stupid/lazy). Some of these questions could raise stereotype threat issues.

I highly recommend changing back to the original activity if possible, or testing out a new version to replace this one very soon.

ChristinaLK commented 5 years ago

Should we just remove it for now?

raynamharris commented 5 years ago

What if we replace the website with a verbal exercise that the instructor read aloud? See https://github.com/carpentries/instructor-training/issues/835. The instructor could recite the following:

"I'm going to the market to buy 16 items: apples, blueberries, cherries, peaches, tea, coffee, beer, wine, cinnamon, nutmeg, basil, oregano, doughnuts, muffins, croissants, bread. What are the 16 items? pause Did anyone notice that these 16 items could be grouped in to four catagories: fruits, beverages, spices, carbs? Did grouping help you remember more words ?"

reedacartwright commented 5 years ago

I think it is worth a shot. At the very least we could use the example temporarily until a better web-app is available.

kariljordan commented 5 years ago

I’m in favor of a verbal exercise. This is what I do. I don’t have them go to any website. I read the words myself and have them write down what they remember.

tobyhodges commented 5 years ago

After some discussion with @fmichonneau and @gvwilson, I will try to develop a small JavaScript app that would run without Flash and replace the existing tool. I'll update on my progress ASAP.

That said, I think it would be great to provide instructions for a verbal equivalent - a "lo-tech" alternative - for those who don't want to/can't use the browser-based exercise. If nothing else, I suspect this would be appreciated from an accessibility perspective.

colinquirk commented 5 years ago

Hi, I just applied to become an instructor so I was flipping through the training. I also happen to be a working memory researcher, so I thought I might be of some help.

I'm not sure how far you have gotten @tobyhodges, but I have a version of a "change detection" task (see main README in the repo) coded in javascript: https://github.com/colinquirk/PsiturkChangeDetection/blob/master/static/js/task.js

It would likely take some modification (and admittedly isn't that well documented), but it might serve as a starting point? I'm also happy to give advice.

I don't have a ton of time right now or I would look into adding something myself. At some point in the future I also plan on writing up some thoughts on this section as a whole and proposing some changes (maybe after actually doing the training :sweat_smile:).

tobyhodges commented 5 years ago

Colin! Some offers are simply too good to pass up 🙂

I've made some progress with the application that I had in mind to test working memory but I've been pretty busy myself recently too. I hope to get a prototype online in the next week, which I would love for you to look at and comment on: the input from an expert in the field will be extremely valuable.

I can't speak for the Carpentries in any official capacity but, in my opinion, it would be great to get your comments on the material in the instructor training curriculum.

colinquirk commented 5 years ago

Great, go ahead and ping me whenever you want and I will let you know my thoughts!

raynamharris commented 5 years ago

I'm just now catching up on this one.

@colinquirk it would be great to hear about experience with the memory lesson following your instructor training. Have you taken the workshop yet?

Given the difficulty @ErinBecker mentioned above https://github.com/carpentries/instructor-training/issues/540#issuecomment-455340719, I have proposed to go back to the original in #915 and added a note about flash in #916. There is a verbal option already, so I think we are okay keeping that as is.

I think this solution will work temporarily while we wait for a new tool from perhaps @colinquirk and/or @tobyhodges.

tobyhodges commented 5 years ago

Thanks Rayna - this is a good solution for now. I've been making some progress with the application but it still needs more attention before I'd be comfortable sharing and asking for feedback.

colinquirk commented 5 years ago

@raynamharris April 4th/5th!

tobyhodges commented 5 years ago

An update on this: I have a prototype working on my local system, but I'm currently stuck at the stage of trying to get the app deployed in a way/place that everyone will be able to access it over the web. I will reach out to some people with more (i.e. any) experience of deploying JS web apps and hopefully have a working version for you to try out and comment on soon.

A brief description of how it works (or doesn't!) in it's current form: after the user presses a "Start" button, the app will flash up twenty words one at a time for a second each. After this has finished, the user can press another button to display the twenty words so that they can see how many they were able to recall correctly. The interface will also provide a dropdown to choose the language of the words that will be displayed. These sets of words in different languages can be contributed via a PR to the GitHub repo, which (I hope) will make our instructor training more accessible across different regions and cultures.

Back in touch again soon, once I've got it running somewhere other than my own laptop.

colinquirk commented 5 years ago

Awesome! I think we could probably host it on the Carpentries site itself right? If we add an html page that sources the JavaScript file I think it would work out alright?

As for the task itself, I have a little bit of feedback! You’ve created what is actually a very popular long term memory task! But 20 is actually way more than you need, in working memory research we usually use 6 items. I know we often talk about 7 plus minus 2 but really it’s more like 3 plus minus 1. The reason people can get up to 7 with digits is because they “chunk” them, which to us is “cheating.” As an example, if I asked you to remember the letters “FBICIA” you probably wouldn’t remember the 6 letters individually, you’d remember the two groups of letters “FBI” and “CIA” (assuming you are in the US at least). So to avoid chunking, I would actually recommend not using words. Instead, how about you select some emojis to use? This way it should work about the same in the underlying code, but it will be a bit harder to chunk (plus we avoid the translating work!). We also usually display everything at once because there are weird effects where people are much better at remembering the first few items and the last few items and they forget the stuff in the middle.

If we want to actually output a score, we can do that in a lot of different ways, we would just need to collect responses from them somehow (and probably have them do it about 10 times or so). You could either show them a single item again and have it be the same or different to what it was before (change detection), you can test a single item and give them two choices to choose between (2 alternative forces choice), or you can test all the items and have multiple choices for all of them (whole report; I’m thinking like a screen of all the possible options off to the side they can click). If we do this I can create histograms similar to what the other program does. I’m not sure if the rest of you feel like getting a score is valuable or if we only need the participants to experience the feeling of trying to remember a lot of things.

Phew! Ok that was a lot of stuff! I think about these things a lot, but in the grand scheme of things they actually don’t matter that much so feel free to ignore all of that!

tobyhodges commented 5 years ago

(Warning: the following contains multiple uses of the word "emoji" as plural...)

It's really helpful to get your feedback, Colin, thanks. I definitely won't be ignoring it :) A few points of response/context, though:

I don't think that we need to collect a score. In the instructor training, we usually ask the trainees to write the number that they remembered in the workshop etherpad/GoogleDoc. Our instructor training workshops tend to be quite pushed for time (as you may have noticed in early April 😉 ), so I don't think that repeating 10x is going to work. As you suggested, what's important is for the trainees to get a feel for the limit of working memory, which they are often surprised to discover is so small. Great to know that twenty items is too many, though.

Regarding the chunking, I have seen this in some workshops recently:

we actually use the FBI/CIA/DNA etc example to describe the concept of chunking, though the examples don't play as well in Europe/elsewhere as they do in North America 🙂
I had a case where one native danish speaker described how having a mother tongue that makes extensive use of compound words helped with remembering a list of words.

I really like the idea of using emoji, as that would remove some of the headache around cultural/regional differences. I would like to present them as quite large and, typing this, I realise that don't actually know if there are limits to the font size in which one can display emoji... Then we could display 8-10 all at once(?), followed by a selection of twenty that include all of those originally displayed. The user would tick boxes/select the emoji that they think were displayed and then submit to get a score. Does that fit better with what you have in mind?

And to finish addressing your points in reverse order... I don't think that we can host on carpentries.org, as that site is built with GitHub Pages, which only works for static pages. The application will need to dynamically render the HTML in response to the user pressing the "Start" button etc.

[edit: feel free to contact me directly if you'd prefer to continue the discussion tbyhdgs at gmail dot com]

karenword commented 5 years ago

I personally like the words in the exercise. When learners get high scores, we discuss their strategies and use that as a segway into chunking. I also second Toby's concerns about creating a time problem -- it's nice and quick as it is. I'm excited to see this happen -- thanks @tobyhodges for your work on creating a tool we can use!

miku commented 5 years ago

May I join the discussion?

I recently attended an instructor training and I did not have Flash installed on my machine (for the short term memory test), so I naturally thought about implementing an alternative. Thankfully, @tobyhodges pointed me to this thread, so I would like to post a minimal alternative to the Flash version here:

https://miku.github.io/activememory/

Currently, it uses a sample of 20 words out of the about thousand most common English words (minus some very short ones), each displayed for 600ms - these values are hard-coded, but adjustable.

It's about 30 lines of JavaScript (plus some generic helpers functions for random sampling, set intersection, ...) and can be hosted as a GitHub Page. It uses a responsive css framework.

There is surely more functionality, which one can implement (statistics, words in different languages, symbols instead of words, ...), but [...]

what's important is for the trainees to get a feel for the limit of working memory, which they are often surprised to discover is so small.

And for that effect, the site may be enough?

raynamharris commented 5 years ago

Wow! The https://miku.github.io/activememory/ site is excellent. I like that you automatically get feedback on the number you correctly or incorrectly remembered. This very nicely illustrates what it's like to be hit with a firehose of information and not be able to remember it all.

raynamharris commented 5 years ago

Thanks everyone for your contributions! Let's try closing this again :)

carpentries / instructor-training

Website to test working memory #540