Automate SE EGRA

[chrisekelley]

Updated LOE here: https://github.com/Tangerine-Community/tangy-form/issues/386

User story: SE EGRA becomes more automated and less hacky. Forms will automatically play sounds and highlight which element triggers the sound - lIke a tangy-helper element.

Tasks:

• [ ] Create tangy-helper custom element that will express the two widgets in the screenshot below - the Question icon and the instruction icon. When a page loads, the audio for these two items queue and play. The student can click on each one multiple times to learn how they work.
• [ ] Add options to tangy-radio-block. (note radio block not buttons) to be able to include sound and images, again being able to queue those to play autmatically and highlight. See example PWA.

[chrisekelley]

When a form loads, it highlights the Question mark image and plays the instructions. It then highlights Big Orange who reads the letter to be clicked. In. the current code, these are in a section called "tangerineMan".

<div style="display:flex;margin-bottom:-2rem;" id="audios">
           <img src="./assets/media/instructions.png" style="max-height:100px;width:auto;border-radius:0.5rem;border:5px solid transparent;margin-right:0.5rem;" onclick="queue('./assets/media/math_84-number-id-question-mark-audio-no-skipping.mp3','instructions');play();" id="instructions">
           <img src="./assets/images/ms-talk2.png" style="max-height:100px;width:auto;border-radius:0.5rem;border:5px solid transparent;" class="talk" onclick="queue('./assets/media/math_1-one.mp3','prompt');play();" id="prompt">
       </div>

Screenshot of example:

[chrisekelley]

Example code:

<tangy-form-item id="sec_num_id_demo" 
            title="Number ID - Demo"
on-open="
         // Make the background blue
         document.styleSheets[1].removeRule(6);
         document.styleSheets[1].addRule('body',`background-color: rgb(247, 246, 246) !important; height: 100% !important; display: flex !important; flex-direction: column !important; justify-content: center !important; align-items: center !important; font-family: Andika, sans-serif !important; font-size: 1.3rem !important; font-weight: 400 !important; line-height: 1.5 !important; margin: 0px !important; color: rgb(33, 33, 33) !important; background-image: url('./images/blue_hills_75.png') !important; background-size: cover !important; background-attachment: fixed !important; background-repeat: no-repeat !important; `,6);
   queue('./assets/media/math_44-number_id_proxy_demo.mp3','instructions')
   // Next two lines obviated by demo video
   // 
   // play();
">
   <tangy-box name="num_id_demo" class="" style="">
       <div style="text-align:center;padding:4rem 1.2rem;background-color:#fff;border-radius:1rem;"> <video autoplay="" style="margin-left: auto" height="400px">
               <source src="./assets/media/math_83-number-id-screenshare-demo-no-skipping.mp4" type="video/mp4"></video> </div>
   </tangy-box>
   <tangy-box name="tangerineMan" class="" style="">
       <div style="display:flex;margin-bottom:-2rem;" id="audios">
           <div id="instructions"></div>
           <div id="prompt"></div>
       </div>
   </tangy-box>
</tangy-form-item>

[chrisekelley]

Letters demo

https://user-images.githubusercontent.com/861535/204327736-45eea137-c796-437a-adfd-c5def24f2c2b.mov

Numbers demo

https://user-images.githubusercontent.com/861535/204330566-c51ba132-3046-455e-a75e-5bd7906e9a17.mov

[chrisekelley]

Specs for new custom element

The new element will display two images with hard-coded path to the image files: ./assets/media/instructions.png and ./assets/images/ms-talk2.png and ids 'instructions' and 'prompt'.

Each image will have an on-click event that will play the sound file.

The new custom element needs the following API:

• pathToInstructionsSound - './assets/media/math_84-number-id-question-mark-audio-no-skipping.mp3
• pathToPromptSound - './assets/media/math_8-eighty.mp3'
• displayInstructions - boolean to enable the Instructions image
• displayPrompt - boolean to enable the prompt image

When the section loads, the sounds should play in-order.

Some sections do not need to display the instructions image and/or play automatically the instructions sound. The 'displayInstructions' boolean will enable.disable this image.

Current implementation

The current queue function takes the following args:

• pathToSound
• id of image to highlight

In the section's on-open event, the play function is usually triggered after the queue function:

   queue('./assets/media/math_8-eighty.mp3','prompt');
   play();;

[zaklearn]

hi, I want to deploy this feature SE EGRA EGMA, in which version is it available and how to do it because there is not much documentation on it.

[lachko]

Instructions on how to install the tools are provided here: https://github.com/ICTatRTI/SE-tools.

[zaklearn]

hi Tangerine-Community/Tangerine, thank you for replying, but unfortunately i'm having some errors while installing the tools. [image: image.png]you can see in here the crash. can you guide me to resolve this?

Le ven. 17 mars 2023 à 09:13, Lachezar Hristov @.***> a écrit :

Instructions on how to install the tools are provided here: https://github.com/ICTatRTI/SE-tools.

— Reply to this email directly, view it on GitHub https://github.com/Tangerine-Community/Tangerine/issues/3473#issuecomment-1473357343, or unsubscribe https://github.com/notifications/unsubscribe-auth/APAU6CBJPKKNNJOODI3J6NDW4QMMPANCNFSM6AAAAAASAA5XZQ . You are receiving this because you commented.Message ID: @.***>

[chrisekelley]

Hi - Would you please re-post the image or paste the text of the error? It did not upload.

[lachko]

@zaklearn you are posting on a feature request to automate the SE EGRA tools. Not related to the actual content set. Please use our support desk for any content related enquiries. Provide us with detailed information on what you are doing, what you are expecting to happen and what is happening.

[zaklearn]

Hi, sure this the error i got.

[chrisekelley]

@zaklearn - Looks like the se-tools directory is not on the filesystem:

If the dir is not in the image, and if you're not mapping the content-sets dir in the container to your server's filesystem, then even if you upload it to the server, it will not appear in the container.

An easy way to get this into the container - look at your startup script - typically, it is start.sh or develop.sh. Look for the directories that are marked with -v or --volume in the run command - for example (from start.sh):

--volume $(pwd)/content-sets:/tangerine/content-sets:delegated \

So, it looks like the content-sets dir is shared, so if you upload the se-tools to that dir, it'll appear inside the container. Plus , it'll remain there even after your restart the container.

[chrisekelley]

Track this feature here: https://github.com/Tangerine-Community/tangy-form/issues/397

[chrisekelley]

Work for this issue is in this branch: https://github.com/Tangerine-Community/Tangerine/tree/3473_media-inputs-automate

Next steps:

• link options to media elements - use the mat-expansion-panel element to create a [panel that drops down under a radio element and displays a list of options. Each of these can be linked to media elements.
• add "Play on Open" checkbox to play the content when the form-item (section) is opened.

[chrisekelley]

Screenshot of Media Input Editor:

[esurface]

New Design

1. The form editor can defined a promptSound and pomptHighlight property for all elements
   • Radio blocks and EFTouch Input sound will be triggered on click
2. The form editor can define the order in which prompt sounds are played in the section of a form
3. The media-input-editor will be removed since the functionality is now at the input level, not the section level

[esurface]

@lachko -- this is a large feature to QA. I'll leave it to you to decide on the best way to do so.