CSUN John Travoltage Manuscript - Remaining To Do's

[emily-phet]

Manuscript

• [x] @briannatomlinson / @kainip Draft Short Abstract (~100-150 words)

• [x] @briannatomlinson / @kainip Please check references in latest version (v5-EM) - there are some needed for supporting docs, and a set of numbered refs that never made it into MLA (these are probably the same as in the ASSETS demo paper).

• [x] (@briannatomlinson / @kainip Create Figure 1 of JT, perhaps two panels (left sim on startup, right sim in action (e.g., lots of electrons being discharged). Alt text needed (can refer to prior pubs including JT for examples of alt text for this sim).

• [x] @terracoda Please check that references in description section (v5-EM) are what you would like, and suggest a reference for prior work in alternative input in the introduction.

• [x] @terracoda Figure 2 - Please confirm that the PDOM figure (blue background indicating static/dynamic/alerts) in the JT description folder is the one you would like included in the paper. Also confirm it is up-to-date (e.g., should it be updated now that the published sim has the sound button in the nav bar). Please write alt text.

• [x] @briannatomlinson Please confirm that numbers in latest draft (v5-EM) include all the participants listed in the spreadsheet in PhET's JT folder. If so, we'll need to add 3 elementary age children and 3 college students w/o VI (these were done by Amanda and Amy and haven't been included to the list of 'earlier interviews'.

• [x] @briannatomlinson with @terracoda Draft Table 2 - Highlights of JT design challenges and some solutions and/or open questions. @terracoda has already put together a list of design challenges in the JT description folder. @briannatomlinson please go through these, add anything from your perspective, draft table with highlights.

• [x] @briannatomlinson @elharden2 (with feedback from @terracoda) Table 3 - data highlights from interviews this paper focuses on. @briannatomlinson can you draft a table, using info you have from the sonlab interviews, and reaching out to @elharden2 for info she has compiled from her interviews?

• [x] @briannatomlinson Please start a doc where we can compile thoughts on insights gained from each population (that we wouldn't have learned if we weren't working across populations - so what did working with group X help us understand that was beneficial for design in general, or for designing for other groups)

Supporting Resources

• [ ] Full documentation of JT’s description (@terracoda has made a version of this, EM to review and give some feedback - I think it needs a few tweaks before being finalized).

• [ ] Link to documentation of sound - @emily-phet to work with @jbphet and AshM to finalize this.

• [ ] @kainip and @elharden2 - Need to put together a single doc with all Qualtrics questions used (sound + description and visuals + sound versions) as supporting doc for paper.

Timeline

• By Nov 25th: Have all of these pieces listed above (refs, tables, figures, supporting resources) finalized (priority for abstract and tables, as these will significantly impact length and content for results/discussion/conclusion text).
• Nov 26-27 EM to draft updates to the results/discussion/conclusion sections of the paper to reflect the content of the new tables. (and cut some length and refs as we'll probably be over)
• Nov 28-30 Final sharing across team for edits.
• Dec 3 Submit (early)!

[briannatomlinson]

@emily-phet do I have access to that JT description folder that you mentioned? (I'm not sure how to get to it, since you've separately shared some of the folders outside of the Sonification Work with me directly). If you have a link that would be sweet!

• I'll take a look at those references. I thought @kainip was going to change them over, I had just copied them in for her.

[briannatomlinson]

Ok. I just finished building out all of the references in Mendeley, and put them into the doc + updated the works cited at the bottom.

@emily-phet I'm currently using MLA version 8 for this (based on the plugin that's available). I couldn't find in the email from CSUN if it was MLA v7 or 8.

• starting a new doc for alt text, tables, etc. based on last years. I'll include them into the folder this is in.

[emily-phet]

@briannatomlinson I just sent an invite to you and @kainip to the "Multimodal Features" folder for John Travoltage. Inside this folder is a sound and description folder...the description folder holds what I mentioned above in the checklist.

[briannatomlinson]

@emily-phet was that spreadsheet supposed to be for JT only?

I only just realized that's where it was... Is there somewhere we should put all of them (the non-JT interviews?)/does that mean there are separate ones for the other sims?

[briannatomlinson]

@kainip - do you know why we only included 5 of the 6 semi-structured interview participants? I just pulled a new copy of the data (and it looks like we have numbers for all 6... so I'm not exactly sure why it wasn't counted before)

[emily-phet]

Maybe one of the later ones didn't have sound (unintended)? There was some issue with one of them, and I just can't recall what it was.

[briannatomlinson]

Yes, that was p605 --- but Prakriti re-ran that session (and there's comments in the semi-structured interview notes about the participants responses to the sounds).

@briannatomlinson to check with Prakriti on this :)

• Just uploaded a new doc (v6)
• Uploaded a folder with docs for alt text, pictures, etc. (in same folder as csun doc)

[elharden2]

I just uploaded Word docs of the two Qualtrics surveys I used into the JT->Multimodal Features->Sound and 2019_CSUN-_Manuscript->Additional Submission Docs folders on the Drive (wasn't sure the best place to put them yet, so feel free to delete them from one folder). @kainip, I think I used the exact same demographics survey questions as you, but it would be a good idea to double check they are indeed the same. I did modify and add to the other Likert-style survey. If you like, you can add any questions I deleted from your survey back into the doc, and we can somehow indicate which questions were just asked to the VI users, and which questions were asked to the sighted learners.

[elharden2]

I realized I probably need to upload a doc with the questions I asked during the semi-structured interviews too, since I modified those a bit from the ones asked to VI users. I'll try and add those to the Drive folder today or tomorrow.

[briannatomlinson]

I started a table drafts doc here today -- https://docs.google.com/document/d/1GPN26b7EFAzUgb4xMytIab5srvG2OhBE4e4MlPfFOrE/edit

[pkaini]

@elharden2 The doc I uploaded has some questions from the semi-structured interview but not all (only the description and sound specific question + end comments). I haven’t had a chance to compare the demographics questions yet, will do by the end of this week.

[briannatomlinson]

Ok! I took a first shot at the tables -- @emily-phet I don't know how relevant you think some of those outcomes are...

@elharden2 I looked through the notes you had on the sounds analysis and the semi-structured interview notes. I think at the level I have concepts broken down to, it seemed like all the students got the concepts. I can check tomorrow or if you have some free time you can review!

If you have other feedback on the outcomes (things we learned from the MS or College students) please add them in! I've just tossed some from the top of my head in there.

• I'm waiting on notes from two of the VI participant sessions then I can finish the counts

[do I check off things at the top if I've "started them" but not finished?]

[briannatomlinson]

I have all the new numbers for the surveys, too (including the last persons data)!

• [x] Needs to get added to the draft but wasn't sure if anyone was looking at it currently (I didn't want to break someone's edits!!)

	new average scores	SD
UMUX	18.67	5.35
BUZZ	71	5.51
BUZZ aesthetics	25.67	2.07
SUS	90	12.25

[terracoda]

@emily-phet, the figure 2 image is almost up-to-date. The Mute Sound button in actually in Sim Resources, but the Keyboard Shortcuts and PhET Menu buttons are not indicated. I will add them.

In addition, do you think the the dynamic descriptions column should indicate what parts of the phrases actually change? For example, I could add braces like in our design documents and code.

[terracoda]

@emily-phet, I updated Figure 2 and uploaded it to the folder. I left both versions, but the one labelled csun-fig02 is the updated one.

Changes:

• I updated the dynamic paragraph text to be what it would be after the discharge as opposed to before.
• I updated the hand position to position negative 1 because with 25 charges, a discharge happens there rather than at position 0.
• I added all the buttons in the Sim Resources Region

If you need a different scenario (e.g. fewer charges) that is easy to change.

Note: I made this one for my poster, so I am not sure if it will fit in the columns of the paper. I we need to adjust the actual document size of the source file to create a figure with different dimension, we will need to call in the help of someone who knows Adobe Illustrator better than me.

[terracoda]

@briannatomlinson, FYI, I adjusted the table that described the sound mappings for the ASSETS demo visual poster. I thought it was important to use the real name (actual labels) for the Foot and Arm in the Sim Features column.

Copying here in case it is at all helpful or needed.

[briannatomlinson]

Awesome, @terracoda! Thanks for the figure! I'll put it into the doc and let you know if it seems too big or anything (I assume it will be fine). I think a scenario with a lot of charges is fine, since that's one half of the image in figure 1

Thanks for the table of the sounds I'll check that too!

[briannatomlinson]

@terracoda Ok, I've updated the figure and table. Thanks a bunch for that!

There's a sentence in the paper that says "Our efforts extend existing practices in image description (Keane and Laverentz) and alternative input (Taliesin’s choice), and methods for designing and evaluating auditory cues (Barrass and Kramer)." <-- I'm wondering if there's a particular citation you would like to add to this?

I've updated the numbers in the paper. Now on v7

[terracoda]

@emily-phet, I updated the John Travoltage's A11y Design document adding a couple more details for the Leg swing slider, updating the PDOM figure, and adding the Keyboard Dialog design and content.

It seems complete from my perspective.

[terracoda]

@emily-phet and @briannatomlinson, I uploaded a word document with a couple of comments on the description section. This was on version 6:

Comments are:

aria-valuetext is used for both the leg (leg swing slider) and the arm (hand position slider). The descriptions for the arm; however, are more dynamic as we use logic to add in directional changes when necessary.

More accurate to say “changes in direction indicating progress towards or away from doorknob.”

I replaced "and" with "indicating"

[briannatomlinson]

Oh! I'll make sure those are integrated into the v7 version -- thanks for letting me know!

[terracoda]

@briannatomlinson, my first comment is to make you and @emily-phet aware of how we used aria-valuetext. If help is needed tweaking the section on descriptions, let me know.

[terracoda]

@briannatomlinson I provided the caption and alternative text for Figure 2. I have uploaded it to the folder on google drive (https://docs.google.com/document/d/1ZuqTGKeIrnsGOxxMP22LWbkqn_DmNzlw3rEd6aZCz1I/edit)

I try to briefly explain the figure for everyone in the caption and then in the alternative text try to fully describe the figure for non-visual readers. I do not try to explain the figure in the alternative text.

If these are too long, let me know and I can try to reduce. Text-filled diagrams are a challenge to fully describe.

[terracoda]

@briannatomlinson, I made comments to Table 2, the Challenges Table regarding descriptions. https://docs.google.com/document/d/1GPN26b7EFAzUgb4xMytIab5srvG2OhBE4e4MlPfFOrE/edit#

I have some concerns, but not sure how much you can fit.

[terracoda]

@briannatomlinson, I commented in the alt text document that I provided new caption and alt text for figure 2.

[terracoda]

@briannatomlinson, I see my name is attached to helping with the Challenges Tables. I'll make some time to incorporate my thoughts, rather than just commenting. Let me know when you are working on this.

[terracoda]

@emily-phet and @briannatomlinson I made a version 2 of Figure 2 which is the the PDOM and Description Strategy. It may be too complicated or too busy, so we can stick withe version one if you think it is better for this paper.

I am posting it here so you can compare to the one in https://github.com/phetsims/a11y-research/issues/123#issuecomment-439021397

Key difference is:

• two versions of the scene summary are shown (page load) and (after interaction)
• due to space I had to angle the alert arrow for the Leg Swing slider

John Travoltage Design folder Both a hi-res PNG and its illustrator file have been uploaded to the John Travoltage Design folder here: https://drive.google.com/drive/folders/1ee85zeZu_V6YzsfmnYoDUnax7cDDXy67

Question Should I put copies of the final figures in the manuscript folder?

[terracoda]

@briannatomlinson and @emily-phet in terms of references in the description section, I commented in version 7 that I think it makes sense to cite my MRP

• Smith, Taliesin L. Access, Action, & Agency: Inclusive Design for the Non-Visual Use of a Highly Interactive Simulation. OCAD University, 15 Apr. 2016, http://openresearch.ocadu.ca/id/eprint/713.

And for alternative input it makes sense to cite the 2018 HCII paper:

• Moore, Emily B., et al. “Keyboard and Screen Reader Accessibility in Complex Interactive Science Simulations: Design Challenges and Elegant Solutions.” Universal Access in Human-Computer Interaction. Methods, Technologies, and Users, edited by Margherita Antona and Constantine Stephanidis, vol. 10907, Springer International Publishing, 2018, pp. 385–400. Crossref, doi:10.1007/978-3-319-92049-8_28.

The above refs are MLA version 8. Is that the correct format?

[terracoda]

@briannatomlinson, FYI, I uploaded version 7 with my references and comments. I am not exactly sure where the latest version is.

[briannatomlinson]

Hi @terracoda thanks for all your notes in the table. I was mostly just brainstorming ideas last week for this. @emily-phet asked me to get a list down so I didn't word things too closely of specifically. I was hoping to talk to her this week about it, but can refine on my own.

If there's particular ones you want worded a certain way, please change them directly. I don't always know all of the terminology you would want to include, or what level of detail is appropriate for a summary vs. a specific description of the interaction/description design. I was trying to give a short description for what things are, using terminology that most people reading the paper would understand.

I'd recommend a more general description or comment and then more specific details (e.g., for the aria roles or design patterns).

We can do what you think is best, though.

[briannatomlinson]

@terracoda thanks for updating the alt-text --> like I said before I don't always know the terminology that you would prefer me to use. I'm almost never involved in the description design so I just don't know the specifics of what things are called.

Please update any of the docs as needed! I'm mostly using place holders so we don't forget to do it for the final submission!

I think using the updated Fig 2 is fine -- would you like to make changes to that description to match it?

Last year CSUN asked for separate documents for: 1) the main manuscript 2) a folder or document of images 3) longer alt text 4) shorter image descriptions

I'm going to update the main documents with the things you've shared (e.g., the alt text, the image descriptions, and the images themselves).

[briannatomlinson]

I made one update to the alt text after switching to your new Parallel DOM figure. @terracoda please check it to make sure you're ok with the changes. https://docs.google.com/document/d/1mVCPlx3SRjKOVcIC-6i3jhwuAtfRdL9ffGN8RGBawUU/edit#heading=h.gjdgxs

[briannatomlinson]

Table updates:

• moved table 1 in there so they were all in one place
• started updating all of the information as suggested edits (following the format Taliesin started!) but then I thought it might be way too hard to follow so I've updated almost all of them and accepted the edits. The goal was to succinctly state what changes we made or actions we took to address the challenges.

Right now, the last column is open challenges, but I think it might be good to have it as an open challenge or successful outcome. If that would be too complex then we can stick to challenges. Thoughts @terracoda, @emily-phet (and @elharden2 or @kainip if you have any!)

[terracoda]

@briannatomlinson, nice work on table. I agree this is kind of hard, and I made some additions and edits.

[terracoda]

@emily-phet and @briannatomlinson, I reworded the description paragraph. It's probably too long now, but hopefully not. Let me know where if any other contributions are needed?

[terracoda]

@emily-phet and @briannatomlinson I shortened some of my text in the challenges table and added my comments to the other tables. I think the tables look good. I hope they will fit. Please let me know if anything else is needed from me. I've provided:

1. references for description section
2. figure 2 - if tweaks are needed just let me know.
3. figure 2 alt text
4. content and comments for tables

[terracoda]

@briannatomlinson and @emily-phet , I added some suggestions for how to describe figure 1 as well. Use what is useful and leave what is not.

[briannatomlinson]

Thanks a bunch! I'll update it this morning before our lab meeting

On Mon, Nov 26, 2018, 6:21 AM Taliesin L. Smith <notifications@github.com wrote:

@briannatomlinson https://github.com/briannatomlinson and @emily-phet https://github.com/emily-phet , I added some suggestions for how to describe figure 1 as well. Use what is useful and leave what is not.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/phetsims/a11y-research/issues/123#issuecomment-441605738, or mute the thread https://github.com/notifications/unsubscribe-auth/AafUyFJ1-zIE9AL8aBtcZ_USSo08PSPmks5uy87DgaJpZM4YbrfS .

[emily-phet]

Remaining to do's

Manuscript

• [x] Last references added (some specific to JT sim need to be included) and final check through all references.

• [x] Final check for length. I think it's probably right at 10 pages.

• [ ] Final read through by team once tables are complete - looking primarily for any errors introduced in editing.

Tables and Figures

• [x] Figure 1 - There's a lot of ceiling and floor showing! Can someone remake these screenshots, with less space above/below John?

• [x] Figure 2 - @terracoda can you tweak this so that the dashed line indicator for the position 2 alert is not angled, but instead has a horizontal break in it (like the solid line indicators elsewhere in the figure)? Whatever is the quickest, easiest way to get the dashed line indicator to not be angled will be good.

• [x] Table 1 - done!

• [x] Table 2 - @emily-phet will come back to this one in the morning.

• [x] Table 3 - done!

• [x] Table 4 - @emily-phet will come back to this one in the morning.

• [ ] alt text and captions - I haven't looked at this, so not sure where they're at. @briannatomlinson let me know if you need any help with these. I'll review them in the morning.

Supporting Resources

• [x] Full documentation of JT’s description (@terracoda has made a version of this, EM to review and give some feedback).

• [ ] Link to documentation of sound - @emily-phet to work with @jbphet and AshM to finalize this.

• [x] @kainip and @elharden2 - Need to put together a single doc with all Qualtrics questions used (sound + description and visuals + sound versions) as supporting doc for paper.

Timeline Goal - materials ready to submit by end of day Friday November 30th

• @emily-phet will try to get table 2 and 3 done Tuesday (Wednesday at latest), and out to everyone for final review of manuscript text, figures and tables.
• Please try to get all references and figures complete by Wednesday.
• Then we can finalize any last tweaks to manuscript (based on review by team) and finalize supporting resources by Friday.

[briannatomlinson]

Emily, would it be useful for me to review the tables at all (like trying to reframe table 4) or would you rather I wait until you look at them to do something?

I can get a new version with less ceiling/floor on the images.

Alt text/captions are available here

[emily-phet]

@briannatomlinson No need to work on the tables today, I'll work on them more tomorrow, and we can talk about them when we meet later.

[terracoda]

Here's a web preview of the updated figure 2. I'll upload the PRINT file to the folder on google drive. I actually didn't know how to make a bent dotted line, but I figured it out. Thanks for asking @emily-phet.

[terracoda]

@briannatomlinson, I inserted the new Fig 2 into your document.

[pkaini]

Need to put together a single doc with all Qualtrics questions used (sound + description and visuals + sound versions) as supporting doc for paper.

@elharden2 i've put all the qualtrics questions in this doc and had a couple of questions:

For the visual usability survey, the instructions read "select from five options (strongly agree, somewhat agree, agree, somewhat disagree, and strongly disagree") but the actual options listed next to the questions have the 7 options, can you confirm which one was used?

In the demographics questions, Q34 (how often do you use high contrast) doesn't show up in the qualtrics survey, although high contrast is listed as one of the options in Q28 (what software tools or other device settings do you use). Was it removed intentionally?

[elharden2]

@pkaini good catch on the visual usability survey questions! That must have been a copy and paste error on my part. I used the 7 options, not 5.

As far as the demographics questions go, I actually didn't modify the demographics survey at all from the one that you originally sent to me, so that question about high contrast must not have been there to begin with (I didn't remove it).

Let me know if you have any other questions!

[pkaini]

Thanks @elharden2 ! I've uploaded the updated version in the additional docs folder (https://drive.google.com/file/d/1CSg90dwMqD08wPcHRbvkWW7_M9xi5FHK/view?usp=sharing) can you take a look and confirm everything is in place?

[elharden2]

@pkaini Just FYI, I added a couple of questions to the semistructured interview that I asked the sighted learners regarding the visuals: What about the visuals in the sim helped you navigate it or figure out what to do? and How do the visuals change as you interact with the sim? I noticed you didn't include all of the semistructured interview questions listed on the Google sheet on the supporting doc, so if these particular questions about the visuals aren't really addressed in the rest of the paper, it's probably not worth listing them on this supporting doc. But, I wanted to give you a heads up just in case. Otherwise, everything looks good-- thanks so much!

[pkaini]

I must have missed them. Thanks for pointing out! I’ll go back and add them to the doc.

[briannatomlinson]

@emily-phet and @terracoda, can you share the links for the two other documents that need to be added?

Ah-ha! It is this one. Ok, thanks Emily! [1. Full documentation of JT’s description (@terracoda has made a version of this, EM to review and give some feedback). Is it this one?]

2. Link to documentation of sound - @emily-phet to work with @jbphet and AshM to finalize this.

[briannatomlinson]

I also went through and updated the links for the references (removed http from links, and updated single pages [tambo, phet design docs, etc.])

+Made bitly's for the links!