Machine Audition -- Scrolling Visualizations

[pmalacho-mit]

Make visuals 'scroll' as new items are added. Still TBD.

Some things to consider:

• When should the visuals scroll? Once 5 (or some #) of notes are added?

[DolevArtzi]

first attempt: https://github.com/mitmedialab/prg-extension-boilerplate/commit/4589a071e4cdc6ba9ae2e4814c6533c828b1f3e0. This commit has 'scrolling' visualizations for the sheet music, as well as 'scrolling' visualizations for waveforms, with the caveat that the colors that notes are being re-drawn as don't seem to match the original color that it was displayed as. Have not worked enough with the frequency/spectrogram visualizations yet.

[DolevArtzi]

somewhat working: https://github.com/mitmedialab/prg-extension-boilerplate/commit/71aedc321239ce41fe1ff5df1d62786899150cf2. This commit fixes two bugs:

• a note outside of the frequency range will get a staff of NaN for sheet music visualization
  • currently, https://github.com/mitmedialab/prg-extension-boilerplate/blob/1cc6b97864cea89afd968653d2c45f669492ee90/packages/scratch-vm/src/extensions/scratch3_musiccreation/sheetmusic.js#L912 and the check just below it get around this by assigning whatever staff is to the extreme of the frequency range that the note is in. This is undesirable in the long run, but suffices as a temporary fix until we figure out what to do with notes outside of the standard sheet music visualization range
    • somehow, adding the set visualization mode to _ with _ block created an issue where magenta gives us some notes with a duration of 0, and in https://github.com/mitmedialab/prg-extension-boilerplate/blob/1cc6b97864cea89afd968653d2c45f669492ee90/packages/scratch-vm/src/extensions/scratch3_musiccreation/sheetmusic.js#L880 that was creating Infinity, causing the loop to never halt

remaining issues:

• (1) it might be worth diagnosing why the above issues come into play with our new visualization block but seemingly weren't encountered or weren't relevant before
• (2) keeping the color consistent with wave visualizations for each note as it gets redrawn in different positions
• (3) add a permanent fix to visualizing notes outside of the range of sheet music visualization

[DolevArtzi]

(2) is mostly fixed by https://github.com/mitmedialab/prg-extension-boilerplate/commit/6ef8f8ac6bbc6de188e3fdfde74a3cdfc5fc0731 aside from a small issue with the legend

edit: (2) fixed by https://github.com/mitmedialab/prg-extension-boilerplate/commit/eced762021cd1b3665f47158dfe6f28b68be99ce

[DolevArtzi]

(3) is fixed by https://github.com/mitmedialab/prg-extension-boilerplate/pull/122, (1) is outdated