Interactive Description Design Tool

[zepumph]

This past week during our yearly team planning, there was a conversation about why sorts of design tools could really improve the workflow for designing interactive description in sims. It was very a very fruitful conversation that I will try to outline here. @jbphet @emily-phet @terracoda @jessegreenberg @zepumph @Matthew-Moore240 @BLFiedler @Ashton-Morris were all in the discussion.

@Matthew-Moore240: Why don’t we simply make a sim that builds the description for us? We give it sliders that can have custom points and then you can add words to those points. We give it buttons you can add phrases too. You wrap that son of gun up in a self voicing wrapper that reads out your custom stuff THEN You have a neat little button that says “export tables and phrases to a google doc” bodabing bodaboom cuts our work in half cuz you’ll have the tables and some phrases all ready to go. EZPZ

• Directly input json strings where possible
• A tool that fills in the A11y View
• SVing tool that let’s us try out the description

---

@zepumph's notes:

Three parts of some tool's implementation

• Hook based on UI component. This is for wiring into a running sim. Later on in conversation we determined that it would likely be best to have a tool that doesn't load a sim at all, but was purely HTML content.
• Dynamic piece - trying out parameters within phrases (basically what we have in the phrase builder, see #141)
• Output tech like aria-live vs aria-valuetext vs webspeech

But there is more than just the dynamic piece, we also want to realize the whole PDOM

Valuable goals (unranked)

• Have the whole PDOM view, so that you can see the context while creating dynamic description.
• Specify dynamic description and play with a sim and hear /see the description change
• Use this tool to limit the complexity of the design doc.
• “Save/load” to come back to work later.
• Better version control and documentation of different feature “branches” or ideas.
• Machine readable output that could be used to put data into the sim?
• “Load” different ideas to give a context about an idea during design meetings.

More ranked:

• Limiting the need for a sim implementation for Iteration
• Limiting the overhead and time of developer/designer back and forth

Current design challenges (unordered)

• Cognitive walkthrough is really time/attention intensive
• The ability to see all pieces of the description in one place (currently design doc puts description all over the place across 50 pages in a google doc)
• Challenge of designing something, and not knowing how it will sound until a dev can put it in the sim.
• Tables take a really long time to create and organize
• Build up of lots of old/out-dated stuff in design doc

Tool goals:

• Quickly put together the pdom
• Define the headings
• Create the Tab order
• Read through the PDOM once it is created
• Insert interactive objects
• Define
• Reduces des doc clutter
• Allow for version control of both state description
• Export PDOM info?????

---

Properties idea from @jessegreenberg

Tool where you can create Properties and your own model that lets you build interactive descriptions. You can create HTML elements and set their content. Elements can be reordered in the DOM. Element content can use Variables. A Variable is dependent on a Property, and you define the strings for the Variable for each Property value. You can create a list of Properties representing a model. When you create a Property you dictate all the values that Property can have. When you create a Variable, you specify the Property it depends on and the strings to be used for each value of that Property. You can then set Property values and watch content update in the example PDOM.

[zepumph]

A discussion with @jessegreenberg and @zepumph yielded the "Properties idea from @jessegreenberg" above. I think that it would be a really cool thing to explore and prototype. I think that a limited example like above could be very useful. I would be interested to see about how we could plug in the current phrase building technology to make this happen. I'll take a look.

[zepumph]

Above is a first pass at mostly adding HTML UI to create Descriptions, Properties, and Elements that can all work with each other. Nothing is working yet, but here is a screen shot at the sort of form that may work at some point.

I think that there is still a layer needed, when creating an element, it isn't enough to just link a single description and property, but instead, a Property is needed for each var in the description. I also need to create a way to implement that mapping. As a first pass, I have only allowed NumberProperties with continuous ranges to occur, so we can just linearly interpolate the range for the valid values in a description variable.

[terracoda]

This is looking really interesting. @jbphet also demoed Rosetta to me and @Matthew-Moore240, and I was curious about how the sim identifies "translatable strings". Some sims like John Travoltage before interactive description was added had no translatable strings.

If designing interactive description after we already have a fully functioning visual design, what if we had a way to tag objects that will need translatable strings, and options to add in translatable strings, e.g., add in headings and paragraphs, and potentially lists.

What you present above may be a similar idea, and may also be more robust. Thinking about how Rosetta currently works and how to extend Rosetta may or may not be helpful.

[Matthew-Moore240]

These mockups look really great! @zepumph

[terracoda]

I want to link this issue to #141 which I still need to review.

[zepumph]

I was thinking this could be a cool thing to experiment with ReactDOM with, I downloaded https://upmostly.com/tutorials/react-hello-world-your-first-react-app into a sandbox and played with it. If we were going to put more investment into this tool, it could be a really nice thing to try to use React for.

[terracoda]

In the mock-up in https://github.com/phetsims/a11y-research/issues/161#issuecomment-757367251

I can imagine multiple cases, like in MAL: Description Name (e.g., mal-dynamic-summary-case-01): Currently, {{lightSource}} light source {{lightSourceOnOff}} directly at {{moleculeName}}. Currently, {{lightSource}} light source {{lightSourceOnOff}} directly at {{moleculeName}}. Currently, {{lightSource}} light source {{lightSourceOnOff}} {{slowSpeedTrue}}} directly at {{moleculeName}}.

Description Name (e.g., mal-dynamic-summary-case-02): Currently, {{{{simPausedTrue}}.}} {{lightSource}} light source {{emits photons}} directly at {{moleculeName}}. Currently, {{{{sim is paused}} {{slowSpeedTrue}}}.}} {{lightSource}}l ight source {{lightSourceOnOff}} directly at {{moleculeName}}.

Description Name (e.g., mal-dynamic-summary-case-03): Currently, {{lightSource}} light source {{lightSourceOnOff}} {{slowSpeedTrue}} directly at {{emptySpaceTrue}}. {{resetHintTrue.}} Currently, {{lightSource}} light source {{lightSourceOnOff}} directly at {{emptySpaceTrue}}. {{resetHintTrue.}} Currently, {{{{simPausedTrue}}.}} {{lightSource}} light source {{lightSourceOnOff}} directly at {{emptySpaceTrue}}. {{resetHintTrue.}}

AND, very simple examples like in OL Description Name (e.g., ol-dynamic-summary): Right now, voltage, V, is {{valueVolts}} volts; resistance, R, is {{valueOhms}} ohms; current, I, is {{valueCurrent}} {{chosenUnit}}.

And then for OL, we 2 other dynamic descriptions: Description Name (e.g., ol-equation-dynamic-description): In equation, letter V is {{comparativeSizeQulatitative}} letter I and {{comparativeSizeQualitative}} to letter R.

Description Name (e.g., ol-circuit-dynamic-description): A pair of wires connect a resistor to a series of batteries. In circuit, batteries supply {{valueVolts}} volts; resistor shows {{amountOfImpuritiesQualitative}};{{sizeOfArrowsQualitative}} indicate a current flowing clockwise at {{valueCurrent}} {{chosenUnits}}.

Note that I made lists into paragraphs with semicolons just as an example.

I'd have to look at properties more to figure out that part.

[terracoda]

My post might provide a few cases for testing.