Histogram sonification

[matthew-blackman]

The sound design team agrees that sonifying the histogram would be a meaningful feature to improve the sim's pedagogy and accessibility. Discussing the histogram learning goals with @catherinecarter, we reached the following specifications that this feature should support:

• This should be considered a pure representation of the histogram, and only infuse information that is available in the histogram.
  • Eg. data that is in the same bin should sound the same.
• While hearing this sound, a user should be able to "hear" the histogram forming as if they were seeing it being drawn on the screen.
• The user should be able to qualitatively distinguish how much data is in each bin, and compare bin heights.
• We do not need to hear individual sounds for each data point, which we feel would sound repetitive with tall bins (especially with large bin widths). It is most important that each histogram sounds internally consistent, and plays for a reasonable length of time (4-5 seconds for the maximum amount of data).
  • We fully support normalizing the duration of the sound so that it is internally consistent for each histogram, but not necessarily proportional across data sets (eg. a histogram with 400 data points does not need a total sound that is 4 times as long as one with 100 data points).
• The user should be able to hear the difference between the way that data is grouped with different bin widths.
• The sonification should be paired with a visual emphasis animating across the corresponding parts of the histogram.
• The sound should be initiated by a UI interaction (ideally a 'play sound' button in the histogram)

[matthew-blackman]

@samreid mentioned that an alternative approach would be to sonify the histogram from bottom to top. His reasoning is that as data is collected, the histogram forms from bottom to top. Ending with the data in the highest bar would also emphasize the mode bin.

As we implement this system, we should keep things as flexible as possible to compare different approaches and keep our options open for experimenting with sound mappings in the histogram.

[matthew-blackman]

@catherinecarter (via Slack):

I’d add, “from left to right” so it’s clear the focus is on how the shape builds from left to right. Also, I’ll admit that when I think of a histogram shape, I think of a Normal distribution as sounding like low to high to low, which would change the mapping as it’s currently being discussed since this would map the height rather than position on the x-axis. It may or may not be a good idea to do that, but I’m starting to picture the sound getting increasing higher, which may or may not lead the user to think it’s an exponential or square root function. I found this: https://www.dailymotion.com/video/x2uemm4, which accentuates the heights of the bars over anything else.

And this:

I found this video and others, all of which are sonifying the y-axis rather than the x-axis: https://www.highcharts.com/demo/highcharts/plotline-context I’m starting to think that the shape is based more on the y-axis (height) of the bars rather than the position on the x-axis. Maybe we can incorporate both?

[catherinecarter]

Also, I made this, which is pretty cool. I didn't know desmos could sonify functions.

https://www.desmos.com/calculator/8au4ygdlum

[matthew-blackman]

Patch from 11/19 pairing:

```diff Subject: [PATCH] Histogram sonification --- Index: js/common/model/HistogramSoundManager.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/common/model/HistogramSoundManager.ts b/js/common/model/HistogramSoundManager.ts new file mode 100644 --- /dev/null (date 1708380081911) +++ b/js/common/model/HistogramSoundManager.ts (date 1708380081911) @@ -0,0 +1,96 @@ +// Copyright 2024, University of Colorado Boulder + +import projectileDataLab from '../../projectileDataLab.js'; +import HistogramData from './HistogramData.js'; +import { Property, TReadOnlyProperty } from '../../../../axon/js/imports.js'; +import { HistogramRepresentation } from './HistogramRepresentation.js'; + +/** + * The HistogramSoundManager class is used to manage the sounds for the histogram in the Projectile Data Lab. + * + * @author Matthew Blackman (PhET Interactive Simulations) + * @author Sam Reid (PhET Interactive Simulations) + */ + +const DURATION_PER_VALUE = 0.1; + +export default class HistogramSoundManager { + + private isPlaying = false; + + private timeSinceStarted = 0; + + private totalSoundDuration = 0; + + private durationForCurrentBin = 0; + + private binnedData = new Map(); + + public constructor( private binWidthProperty: TReadOnlyProperty, + private visualizationProperty: Property ) { + + // Set up listeners for the binWidthProperty and visualizationProperty to cancel any ongoing sound + + // Also, if the data changes, we need to cancel any ongoing sound + } + + public setHistogramData( data: HistogramData[] ): void { + + const binnedData = new Map(); + const binWidth = this.binWidthProperty.value; + + for ( let i = 0; i < data.length; i++ ) { + const projectile = data[ i ]; + + // Calculate the bin for this value by its lower bound + const bin = Math.floor( projectile.x / binWidth ) * binWidth; + + // Update the count for this bin + const binCount = ( binnedData.get( bin ) || 0 ) + 1; + binnedData.set( bin, binCount ); + } + + this.binnedData = binnedData; + + this.totalSoundDuration = DURATION_PER_VALUE * data.length; + } + + public playHistogramSound(): void { + this.timeSinceStarted = 0; + this.isPlaying = true; + + const sortedBins = Array.from( this.binnedData.keys() ).sort( ( a, b ) => a - b ); + + const firstBinValue = sortedBins[ 0 ] + this.binWidthProperty.value / 2; + const firstBinCount = this.binnedData.get( sortedBins[ 0 ] ) || 0; + + const soundFrequency = this.frequencyForDistance( firstBinValue ); + this.durationForCurrentBin = DURATION_PER_VALUE * firstBinCount; + + // Start playing the continuous sound generator + } + + public step( dt: number ): void { + + if ( this.isPlaying ) { + + this.timeSinceStarted += dt; + + // If we have completed the sound for the previous bin, adjust the frequency of the sound generator for the next bin + + + // If we have completed the sound for the last bin, stop playing + if ( this.timeSinceStarted > this.totalSoundDuration ) { + this.isPlaying = false; + // Stop the sound + } + } + } + + private frequencyForDistance( distance: number ): number { + // Map the distance to a frequency + return 400 + distance * 100; + } +} + +projectileDataLab.register( 'HistogramSoundManager', HistogramSoundManager ); \ No newline at end of file Index: js/common/model/Histogram.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/common/model/Histogram.ts b/js/common/model/Histogram.ts --- a/js/common/model/Histogram.ts (revision 7940a9dffaaf4a35dcb496ef5e9c46f016ecf3af) +++ b/js/common/model/Histogram.ts (date 1708380557948) @@ -13,6 +13,7 @@ import { BIN_STRATEGY_PROPERTY } from '../PDLQueryParameters.js'; import PDLConstants from '../PDLConstants.js'; import StringUnionProperty from '../../../../axon/js/StringUnionProperty.js'; +import HistogramSoundManager from './HistogramSoundManager.js'; /** * The Histogram class is used to represent the histogram in the Projectile Data Lab. @@ -114,6 +115,8 @@ phetioFeatured: true, phetioDocumentation: 'This property indicates whether the histogram is showing bars (one per bin) or blocks (one per projectile).' } ); + + const histogramSoundManager = new HistogramSoundManager( this.binWidthProperty, this.representationProperty ); } public reset(): void { Index: js/common/view/HistogramCanvasPainter.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/common/view/HistogramCanvasPainter.ts b/js/common/view/HistogramCanvasPainter.ts --- a/js/common/view/HistogramCanvasPainter.ts (revision 7940a9dffaaf4a35dcb496ef5e9c46f016ecf3af) +++ b/js/common/view/HistogramCanvasPainter.ts (date 1708380799959) @@ -26,6 +26,12 @@ private data: HistogramData[] = []; private selectedData: HistogramData | null = null; + // Which column is currently being sonified, or null if none + private sonifiedColumn: number | null = null; + + // The data that is currently being sonified + private sonifiedData: HistogramData | null = null; + public constructor( private readonly chartTransform: ChartTransform, private readonly binWidthProperty: TReadOnlyProperty, private readonly histogramRepresentationProperty: TReadOnlyProperty, @@ -44,6 +50,14 @@ this.selectedData = selectedData; } + /** + * Sets the data that is currently being sonified and redraws the plot. + */ + public setDataSonification( sonifiedColumn: number | null, sonifiedData: HistogramData | null ): void { + this.sonifiedColumn = sonifiedColumn; + this.sonifiedData = sonifiedData; + } + public paintCanvas( context: CanvasRenderingContext2D ): void { const histogramRepresentation = this.histogramRepresentationProperty.value; @@ -116,6 +130,12 @@ context.stroke(); } + if ( this.sonifiedColumn !== null && this.sonifiedData !== null ) { + // Draw a rectangle overlay for the sonified column + + // Draw a block over the sonified data value + } + context.restore(); } }

[samreid]

Updated patch from collaboration with @matthew-blackman. Please bring the sound files from CAV over too (not in the patch):

```diff Subject: [PATCH] Add assertion message, see https://github.com/phetsims/projectile-data-lab/issues/43 --- Index: js/sources/view/SourcesScreenView.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/sources/view/SourcesScreenView.ts b/js/sources/view/SourcesScreenView.ts --- a/js/sources/view/SourcesScreenView.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/sources/view/SourcesScreenView.ts (date 1708382181463) @@ -52,12 +52,14 @@ const createHistogramNode = ( node: Node ) => new HistogramNode( model.fieldProperty, model.fields, - model.histogram.zoomProperty, - model.histogram.binWidthProperty, - model.histogram.representationProperty, ProjectileDataLabStrings.distanceStringProperty, - model.histogram.selectedBinWidthProperty, - model.histogram.selectedTotalBinsProperty, + model.histogram, + // model.histogram.zoomProperty, + // model.histogram.binWidthProperty, + // model.histogram.representationProperty, + + // model.histogram.selectedBinWidthProperty, + // model.histogram.selectedTotalBinsProperty, node, PDLColors.histogramDataFillColorProperty, PDLColors.histogramDataStrokeColorProperty, { Index: js/measures/view/MeasuresHistogramNode.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/measures/view/MeasuresHistogramNode.ts b/js/measures/view/MeasuresHistogramNode.ts --- a/js/measures/view/MeasuresHistogramNode.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/measures/view/MeasuresHistogramNode.ts (date 1708382287806) @@ -23,6 +23,7 @@ import Utils from '../../../../dot/js/Utils.js'; import PDLConstants from '../../common/PDLConstants.js'; import NumberProperty from '../../../../axon/js/NumberProperty.js'; +import Histogram from '../../common/model/Histogram.js'; /** * The measures histogram node is a histogram node that also shows the mean and standard deviation of the data. @@ -38,9 +39,10 @@ public constructor( fieldProperty: TReadOnlyProperty, fields: Field[], - zoomProperty: NumberProperty, - binWidthProperty: TReadOnlyProperty, - histogramRepresentationProperty: Property, + histogram: Histogram, + // zoomProperty: NumberProperty, + // binWidthProperty: TReadOnlyProperty, + // histogramRepresentationProperty: Property, horizontalAxisLabelText: TReadOnlyProperty, isMeanVisibleProperty: BooleanProperty, isStandardDeviationVisibleProperty: BooleanProperty, @@ -50,19 +52,20 @@ standardErrorProperty: PhetioProperty, intervalTool: IntervalTool, intervalToolVisibleProperty: TReadOnlyProperty, - selectedBinWidthProperty: Property, - selectedTotalBinsProperty: Property, + // selectedBinWidthProperty: Property, + // selectedTotalBinsProperty: Property, comboBoxParent: Node, options: MeasuresHistogramNodeOptions ) { super( fieldProperty, fields, - zoomProperty, - binWidthProperty, - histogramRepresentationProperty, horizontalAxisLabelText, - selectedBinWidthProperty, - selectedTotalBinsProperty, + histogram, + // zoomProperty, + // binWidthProperty, + // histogramRepresentationProperty, + // selectedBinWidthProperty, + // selectedTotalBinsProperty, comboBoxParent, PDLColors.histogramDataFillColorProperty, Index: js/variability/view/VariabilityScreenView.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/variability/view/VariabilityScreenView.ts b/js/variability/view/VariabilityScreenView.ts --- a/js/variability/view/VariabilityScreenView.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/variability/view/VariabilityScreenView.ts (date 1708382181453) @@ -50,12 +50,13 @@ const createHistogramNode = ( node: Node ) => new HistogramNode( model.fieldProperty, model.fields, - model.histogram.zoomProperty, - model.histogram.binWidthProperty, - model.histogram.representationProperty, ProjectileDataLabStrings.distanceStringProperty, - model.histogram.selectedBinWidthProperty, - model.histogram.selectedTotalBinsProperty, + // model.histogram.zoomProperty, + // model.histogram.binWidthProperty, + // model.histogram.representationProperty, + // model.histogram.selectedBinWidthProperty, + // model.histogram.selectedTotalBinsProperty, + model.histogram, node, PDLColors.histogramDataFillColorProperty, PDLColors.histogramDataStrokeColorProperty, { Index: js/common-vsm/model/VSMModel.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/common-vsm/model/VSMModel.ts b/js/common-vsm/model/VSMModel.ts --- a/js/common-vsm/model/VSMModel.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/common-vsm/model/VSMModel.ts (date 1708383020516) @@ -226,8 +226,10 @@ this.fieldProperty.value.launchProjectile(); } - public step( dt: number ): void { + public override step( dt: number ): void { + super.step( dt ); + // TODO if ( !this.isPlayingProperty.value ) { return; } Index: js/common/model/PDLModel.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/common/model/PDLModel.ts b/js/common/model/PDLModel.ts --- a/js/common/model/PDLModel.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/common/model/PDLModel.ts (date 1708382959251) @@ -172,6 +172,10 @@ } ); } + public step( dt: number ): void { + this.histogram.step( dt ); + } + public abstract launchButtonPressed(): void; public reset(): void { Index: js/common/view/HistogramNode.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/common/view/HistogramNode.ts b/js/common/view/HistogramNode.ts --- a/js/common/view/HistogramNode.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/common/view/HistogramNode.ts (date 1708385121634) @@ -19,22 +19,20 @@ import SamplingField from '../../sampling/model/SamplingField.js'; import Tandem from '../../../../tandem/js/Tandem.js'; import VSMField from '../../common-vsm/model/VSMField.js'; -import { HistogramRepresentation } from '../model/HistogramRepresentation.js'; import PDLText from './PDLText.js'; import ProjectileDataLabStrings from '../../ProjectileDataLabStrings.js'; import projectileDataLab from '../../projectileDataLab.js'; import isSettingPhetioStateProperty from '../../../../tandem/js/isSettingPhetioStateProperty.js'; import ABSwitch from '../../../../sun/js/ABSwitch.js'; import HistogramRepresentationIconNode from './HistogramRepresentationIconNode.js'; -import Property from '../../../../axon/js/Property.js'; import BinControlNode from './BinControlNode.js'; import TickMarkSet from '../../../../bamboo/js/TickMarkSet.js'; -import { ZOOM_LEVELS } from '../model/Histogram.js'; -import NumberProperty from '../../../../axon/js/NumberProperty.js'; +import Histogram, { ZOOM_LEVELS } from '../model/Histogram.js'; import pdlToggleButtonA_mp3 from '../../../sounds/pdlToggleButtonA_mp3.js'; import pdlToggleButtonB_mp3 from '../../../sounds/pdlToggleButtonB_mp3.js'; import SoundClip from '../../../../tambo/js/sound-generators/SoundClip.js'; import soundManager from '../../../../tambo/js/soundManager.js'; +import { TextPushButton } from '../../../../sun/js/imports.js'; /** * Shows the Histogram in the Projectile Data Lab simulation. @@ -63,12 +61,13 @@ public constructor( fieldProperty: TReadOnlyProperty, fields: Field[], - zoomProperty: NumberProperty, - binWidthProperty: TReadOnlyProperty, - histogramRepresentationProperty: Property, + // histogram.zoomProperty: NumberProperty, horizontalAxisLabelText: TReadOnlyProperty, - selectedBinWidthProperty: Property, - selectedTotalBinsProperty: Property, + histogram: Histogram, + // binWidthProperty: TReadOnlyProperty, + // histogramRepresentationProperty: Property, + // selectedBinWidthProperty: Property, + // selectedTotalBinsProperty: Property, comboBoxParent: Node, blockFillProperty: ColorProperty, blockStrokeProperty: ColorProperty, @@ -99,7 +98,7 @@ stroke: 'black' } ); - const histogramPainter = new HistogramCanvasPainter( this.chartTransform, binWidthProperty, histogramRepresentationProperty, + const histogramPainter = new HistogramCanvasPainter( this.chartTransform, histogram.binWidthProperty, histogram.representationProperty, blockFillProperty, blockStrokeProperty ); // Grid lines along the y-axis (the lines themselves are horizontal). Changes based on the zoom level @@ -173,7 +172,7 @@ ] } ); - binWidthProperty.link( binWidth => { + histogram.binWidthProperty.link( binWidth => { horizontalAxisGridLines.setSpacing( binWidth ); chartCanvasNode.update(); } ); @@ -182,11 +181,11 @@ chartCanvasNode.update(); } ); - histogramRepresentationProperty.link( () => { + histogram.representationProperty.link( () => { chartCanvasNode.update(); } ); - const zoomButtonGroup = new PlusMinusZoomButtonGroup( zoomProperty, { + const zoomButtonGroup = new PlusMinusZoomButtonGroup( histogram.zoomProperty, { tandem: options.tandem.createTandem( 'zoomButtonGroup' ), orientation: 'vertical', centerY: this.chartTransform.viewHeight, @@ -225,15 +224,19 @@ const field = fieldProperty.value; if ( field instanceof VSMField ) { histogramPainter.setHistogramData( fieldProperty.value.landedProjectiles, field.selectedProjectileProperty.value ); + histogram.histogramSoundManager.setHistogramData( fieldProperty.value.landedProjectiles ); } else if ( field instanceof SamplingField ) { const samples = field.getHistogramData(); const selectedSampleNumber = field.selectedSampleNumberProperty.value; histogramPainter.setHistogramData( samples, samples[ selectedSampleNumber - 1 ] ); + + histogram.histogramSoundManager.setHistogramData( samples ); } else { assert && assert( false, 'unhandled field type' ); } + chartCanvasNode.update(); } }; @@ -268,19 +271,19 @@ // When the field or bin width changes, redraw the histogram fieldProperty.link( () => updateHistogram() ); - binWidthProperty.link( () => updateHistogram() ); - zoomProperty.link( () => { + histogram.binWidthProperty.link( () => updateHistogram() ); + histogram.zoomProperty.link( () => { - const maxCount = ZOOM_LEVELS[ zoomProperty.value ].maxCount; + const maxCount = ZOOM_LEVELS[ histogram.zoomProperty.value ].maxCount; this.chartTransform.setModelYRange( new Range( 0, maxCount ) ); - const tickSpacing = ZOOM_LEVELS[ zoomProperty.value ].maxCount / 5; + const tickSpacing = ZOOM_LEVELS[ histogram.zoomProperty.value ].maxCount / 5; verticalTickLabelSet.setSpacing( tickSpacing ); verticalTickMarkSet.setSpacing( tickSpacing ); majorVerticalAxisGridLines.setSpacing( tickSpacing ); - const spacing = ZOOM_LEVELS[ zoomProperty.value ].minorSpacing; + const spacing = ZOOM_LEVELS[ histogram.zoomProperty.value ].minorSpacing; if ( spacing !== null ) { verticalAxisGridLines.setSpacing( spacing ); } @@ -289,7 +292,7 @@ updateHistogram(); } ); - const binControlNode = new BinControlNode( comboBoxParent, selectedBinWidthProperty, selectedTotalBinsProperty, { + const binControlNode = new BinControlNode( comboBoxParent, histogram.selectedBinWidthProperty, histogram.selectedTotalBinsProperty, { tandem: options.tandem.createTandem( 'binControlNode' ), leftTop: this.chartNode.leftBottom.plusXY( 15, CHART_UI_MARGIN ), visiblePropertyOptions: { @@ -299,7 +302,7 @@ this.addChild( binControlNode ); const barBlockSwitch = new ABSwitch( - histogramRepresentationProperty, + histogram.representationProperty, 'blocks', new HistogramRepresentationIconNode( blockFillProperty, blockStrokeProperty, 'blocks' ), 'bars', new HistogramRepresentationIconNode( blockFillProperty, blockStrokeProperty, 'bars' ), { tandem: options.tandem.createTandem( 'barBlockSwitch' ), @@ -313,6 +316,15 @@ } ); this.addChild( barBlockSwitch ); + const playHistogramSoundButton = new TextPushButton( 'Play Sound', { + tandem: options.tandem.createTandem( 'playHistogramSoundButton' ), + listener: () => { + histogram.histogramSoundManager.playHistogramSound(); + console.log( 'play sound' ); + } + } ); + this.addChild( playHistogramSoundButton ); + ManualConstraint.create( this, [ this.chartNode, this.chartBackground, horizontalAxisLabel ], ( chartNodeProxy, chartBackgroundProxy, horizontalAxisLabelProxy ) => { horizontalAxisLabelProxy.centerX = chartBackgroundProxy.centerX; horizontalAxisLabelProxy.top = chartNodeProxy.bottom + CHART_UI_MARGIN; @@ -327,6 +339,7 @@ zoomButtonGroup, binControlNode, barBlockSwitch, + playHistogramSoundButton, this.chartNode ]; } Index: js/sampling/view/SamplingAccordionBox.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/sampling/view/SamplingAccordionBox.ts b/js/sampling/view/SamplingAccordionBox.ts --- a/js/sampling/view/SamplingAccordionBox.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/sampling/view/SamplingAccordionBox.ts (date 1708382472387) @@ -15,6 +15,7 @@ import SamplingHistogramNode from './SamplingHistogramNode.js'; import Launcher from '../../common/model/Launcher.js'; import NumberProperty from '../../../../axon/js/NumberProperty.js'; +import Histogram from '../../common/model/Histogram.js'; /** * The SamplingAccordionBox is an accordion UI component for the Projectile Data Lab simulation. @@ -33,14 +34,15 @@ public readonly bottomThumbnailNode: SampleSizeThumbnailNode; public constructor( + histogram: Histogram, launcherProperty: TReadOnlyProperty, sampleSizeProperty: TReadOnlyProperty, numberOfSamplesProperty: TReadOnlyProperty, fieldProperty: TReadOnlyProperty, fields: SamplingField[], zoomProperty: NumberProperty, - selectedBinWidthProperty: Property, - selectedTotalBinsProperty: Property, + // selectedBinWidthProperty: Property, + // selectedTotalBinsProperty: Property, binWidthProperty: TReadOnlyProperty, comboBoxParent: Node, histogramRepresentationProperty: Property, @@ -53,12 +55,13 @@ numberOfSamplesProperty, fieldProperty, fields, - zoomProperty, - binWidthProperty, - histogramRepresentationProperty, + histogram, + // zoomProperty, + // binWidthProperty, + // histogramRepresentationProperty, ProjectileDataLabStrings.meanDistanceStringProperty, - selectedBinWidthProperty, - selectedTotalBinsProperty, + // selectedBinWidthProperty, + // selectedTotalBinsProperty, comboBoxParent, clearCurrentField, { tandem: providedOptions.tandem.createTandem( 'histogramNode' ) Index: js/common/view/HistogramCanvasPainter.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/common/view/HistogramCanvasPainter.ts b/js/common/view/HistogramCanvasPainter.ts --- a/js/common/view/HistogramCanvasPainter.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/common/view/HistogramCanvasPainter.ts (date 1708381567253) @@ -26,6 +26,12 @@ private data: HistogramData[] = []; private selectedData: HistogramData | null = null; + // Which column is currently being sonified, or null if none + private sonifiedColumn: number | null = null; + + // The data that is currently being sonified + private sonifiedData: HistogramData | null = null; + public constructor( private readonly chartTransform: ChartTransform, private readonly binWidthProperty: TReadOnlyProperty, private readonly histogramRepresentationProperty: TReadOnlyProperty, @@ -44,6 +50,14 @@ this.selectedData = selectedData; } + /** + * Sets the data that is currently being sonified and redraws the plot. + */ + public setDataSonification( sonifiedColumn: number | null, sonifiedData: HistogramData | null ): void { + this.sonifiedColumn = sonifiedColumn; + this.sonifiedData = sonifiedData; + } + public paintCanvas( context: CanvasRenderingContext2D ): void { const histogramRepresentation = this.histogramRepresentationProperty.value; @@ -116,6 +130,12 @@ context.stroke(); } + if ( this.sonifiedColumn !== null && this.sonifiedData !== null ) { + // Draw a rectangle overlay for the sonified column + + // Draw a block over the sonified data value + } + context.restore(); } } Index: js/sampling/model/SamplingModel.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/sampling/model/SamplingModel.ts b/js/sampling/model/SamplingModel.ts --- a/js/sampling/model/SamplingModel.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/sampling/model/SamplingModel.ts (date 1708383020525) @@ -216,7 +216,8 @@ } } - public step( dt: number ): void { + public override step( dt: number ): void { + super.step( dt ); if ( !this.isPlayingProperty.value ) { return; Index: js/sampling/view/SamplingScreenView.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/sampling/view/SamplingScreenView.ts b/js/sampling/view/SamplingScreenView.ts --- a/js/sampling/view/SamplingScreenView.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/sampling/view/SamplingScreenView.ts (date 1708382504778) @@ -143,6 +143,7 @@ this.behindProjectilesLayer.addChild( this.fieldSignNode ); this.accordionBox = new SamplingAccordionBox( + model.histogram, model.launcherProperty, model.sampleSizeProperty, model.numberOfCompletedSamplesProperty, @@ -150,9 +151,9 @@ model.fieldProperty, model.fields, model.histogram.zoomProperty, - model.histogram.selectedBinWidthProperty, - model.histogram.selectedTotalBinsProperty, model.histogram.binWidthProperty, + // model.histogram.selectedTotalBinsProperty, + // model.histogram.binWidthProperty, this, model.histogram.representationProperty, () => model.clearCurrentField(), { Index: js/sampling/view/SamplingHistogramNode.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/sampling/view/SamplingHistogramNode.ts b/js/sampling/view/SamplingHistogramNode.ts --- a/js/sampling/view/SamplingHistogramNode.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/sampling/view/SamplingHistogramNode.ts (date 1708382287801) @@ -20,6 +20,7 @@ import NumberProperty from '../../../../axon/js/NumberProperty.js'; import { Multilink } from '../../../../axon/js/imports.js'; import PDLQueryParameters from '../../common/PDLQueryParameters.js'; +import Histogram from '../../common/model/Histogram.js'; /** * The SamplingHistogramNode shows the histogram for a sampling field, extending the standard HistogramNode and adding @@ -33,24 +34,27 @@ numberOfSamplesProperty: TReadOnlyProperty, fieldProperty: TReadOnlyProperty, fields: Field[], - zoomProperty: NumberProperty, - binWidthProperty: TReadOnlyProperty, - histogramRepresentationProperty: Property, + histogram: Histogram, + // zoomProperty: NumberProperty, + // binWidthProperty: TReadOnlyProperty, + // histogramRepresentationProperty: Property, horizontalAxisLabelText: TReadOnlyProperty, - selectedBinWidthProperty: Property, - selectedTotalBinsProperty: Property, + // selectedBinWidthProperty: Property, + // selectedTotalBinsProperty: Property, comboBoxParent: Node, clearCurrentField: () => void, options: HistogramNodeOptions ) { super( fieldProperty, fields, - zoomProperty, - binWidthProperty, - histogramRepresentationProperty, horizontalAxisLabelText, - selectedBinWidthProperty, - selectedTotalBinsProperty, + histogram, + // zoomProperty, + // binWidthProperty, + // histogramRepresentationProperty, + + // selectedBinWidthProperty, + // selectedTotalBinsProperty, comboBoxParent, PDLColors.meanMarkerFillProperty, PDLColors.meanMarkerStrokeProperty, Index: js/common/model/Histogram.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/common/model/Histogram.ts b/js/common/model/Histogram.ts --- a/js/common/model/Histogram.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/common/model/Histogram.ts (date 1708383096518) @@ -13,6 +13,7 @@ import { BIN_STRATEGY_PROPERTY } from '../PDLQueryParameters.js'; import PDLConstants from '../PDLConstants.js'; import StringUnionProperty from '../../../../axon/js/StringUnionProperty.js'; +import HistogramSoundManager from './HistogramSoundManager.js'; /** * The Histogram class is used to represent the histogram in the Projectile Data Lab. @@ -59,6 +60,7 @@ export default class Histogram { + // Bin width represents the distance between adjacent field lines. It also affects how data is grouped for the histogram. // The prefix 'selected' means it is the value selected by the user, and may differ from the displayed bin width // depending on the BIN_STRATEGY_PROPERTY. @@ -76,6 +78,7 @@ public readonly binWidthProperty: TReadOnlyProperty; public readonly representationProperty: Property; + public readonly histogramSoundManager: HistogramSoundManager; public constructor( tandem: Tandem, providedOptions: HistogramOptions ) { this.selectedBinWidthProperty = new Property( 1, { @@ -114,6 +117,11 @@ phetioFeatured: true, phetioDocumentation: 'This property indicates whether the histogram is showing bars (one per bin) or blocks (one per projectile).' } ); + + this.histogramSoundManager = new HistogramSoundManager( this.binWidthProperty, this.representationProperty ); + } + public step( dt: number ): void { + this.histogramSoundManager.step( dt ); } public reset(): void { Index: js/measures/view/MeasuresScreenView.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/measures/view/MeasuresScreenView.ts b/js/measures/view/MeasuresScreenView.ts --- a/js/measures/view/MeasuresScreenView.ts (revision 474de904262dd324b840782cc8fefd516034293c) +++ b/js/measures/view/MeasuresScreenView.ts (date 1708382472393) @@ -58,9 +58,10 @@ const createHistogramNode = ( comboBoxParent: Node ) => new MeasuresHistogramNode( model.fieldProperty, model.fields, - model.histogram.zoomProperty, - model.histogram.binWidthProperty, - model.histogram.representationProperty, + model.histogram, + // model.histogram.zoomProperty, + // model.histogram.binWidthProperty, + // model.histogram.representationProperty, ProjectileDataLabStrings.distanceStringProperty, model.isMeanVisibleProperty, model.isStandardDeviationVisibleProperty, @@ -70,8 +71,8 @@ model.standardErrorDistanceProperty, model.intervalTool, model.isIntervalToolVisibleProperty, - model.histogram.selectedBinWidthProperty, - model.histogram.selectedTotalBinsProperty, + // model.histogram.selectedBinWidthProperty, + // model.histogram.selectedTotalBinsProperty, comboBoxParent, { tandem: options.tandem.createTandem( histogramAccordionBoxTandemName ).createTandem( 'histogramNode' ) } ); Index: js/common/model/HistogramSoundManager.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/js/common/model/HistogramSoundManager.ts b/js/common/model/HistogramSoundManager.ts new file mode 100644 --- /dev/null (date 1708385218175) +++ b/js/common/model/HistogramSoundManager.ts (date 1708385218175) @@ -0,0 +1,154 @@ +// Copyright 2024, University of Colorado Boulder + +import projectileDataLab from '../../projectileDataLab.js'; +import HistogramData from './HistogramData.js'; +import { BooleanProperty, NumberProperty, Property, TReadOnlyProperty } from '../../../../axon/js/imports.js'; +import { HistogramRepresentation } from './HistogramRepresentation.js'; +import ContinuousPropertySoundClip from '../../../../tambo/js/sound-generators/ContinuousPropertySoundClip.js'; +import Range from '../../../../dot/js/Range.js'; + +import intervalToolLoop_wav from '../../../sounds/intervalToolLoop_wav.js'; +import soundManager from '../../../../tambo/js/soundManager.js'; +import DerivedProperty from '../../../../axon/js/DerivedProperty.js'; + +/** + * The HistogramSoundManager class is used to manage the sounds for the histogram in the Projectile Data Lab. + * + * @author Matthew Blackman (PhET Interactive Simulations) + * @author Sam Reid (PhET Interactive Simulations) + */ + +const DURATION_PER_VALUE = 0.1; + +export default class HistogramSoundManager { + + private isPlaying = false; + + // private timeSinceStarted = 0; + + private totalSoundDuration = 0; + + private durationForCurrentBin = 0; + + private binnedData = new Map(); + private readonly continuousPropertySoundGenerator: ContinuousPropertySoundClip; + private readonly dataValueProperty: NumberProperty; + private timeRemainingInCurrentBin: number; + private currentBinIndex: number = 0; + + public constructor( private binWidthProperty: TReadOnlyProperty, + private visualizationProperty: Property ) { + + // Set up listeners for the binWidthProperty and visualizationProperty to cancel any ongoing sound + + // Also, if the data changes, we need to cancel any ongoing sound + this.dataValueProperty = new NumberProperty( 50 ); + this.continuousPropertySoundGenerator = new ContinuousPropertySoundClip( this.dataValueProperty, new Range( 1, 100 ), intervalToolLoop_wav, { + initialOutputLevel: 0.25, + playbackRateCenterOffset: 0, + + enableControlProperties: [ new BooleanProperty( true ) ], + trimSilence: false, // a very precise sound file is used, so make sure it doesn't get changed + fadeTime: 0.3, + delayBeforeStop: 0.25, + playbackRateSpanOctaves: 1.5, + stopOnDisabled: true + // additionalAudioNodes: [ + // biquadFilterNode + // ] + } ); + + soundManager.addSoundGenerator( this.continuousPropertySoundGenerator, { + // associatedViewNode + } ); + } + + public setHistogramData( data: HistogramData[] ): void { + + const binnedData = new Map(); + const binWidth = this.binWidthProperty.value; + + for ( let i = 0; i < data.length; i++ ) { + const projectile = data[ i ]; + + // Calculate the bin for this value by its lower bound + const bin = Math.floor( projectile.x / binWidth ) * binWidth; + + // Update the count for this bin + const binCount = ( binnedData.get( bin ) || 0 ) + 1; + binnedData.set( bin, binCount ); + } + + this.binnedData = binnedData; + + this.totalSoundDuration = DURATION_PER_VALUE * data.length; + + console.log( 'binned data changed: ', this.binnedData ); + } + + public playHistogramSound(): void { + // this.timeSinceStarted = 0; + this.isPlaying = true; + + // Start playing the continuous sound generator + this.continuousPropertySoundGenerator.play(); + + console.log( 'Playing sound at frequency:, this.isPlaying = ' + this.isPlaying ); + const sortedBins = Array.from( this.binnedData.keys() ).sort( ( a, b ) => a - b ); + + // const firstBinValue = sortedBins[ 0 ] + this.binWidthProperty.value / 2; + const firstBinCount = this.binnedData.get( sortedBins[ 0 ] ) || 0; + + this.timeRemainingInCurrentBin = DURATION_PER_VALUE * firstBinCount; + + this.currentBinIndex = 0; + + } + + public step( dt: number ): void { + + + if ( this.isPlaying ) { + + // this.timeSinceStarted += dt; + this.timeRemainingInCurrentBin -= dt; + + if ( this.timeRemainingInCurrentBin <= 0 ) { + this.currentBinIndex++; + + // if we went past the edge of the bins, stop playing + if ( this.currentBinIndex >= this.binnedData.size ) { + this.isPlaying = false; + return; + } + } + + const sortedBins = Array.from( this.binnedData.keys() ).sort( ( a, b ) => a - b ); + const binCount = this.binnedData.get( sortedBins[ this.currentBinIndex ] ) || 0; + // const soundFrequency = this.frequencyForDistance( firstBinValue ); + this.durationForCurrentBin = DURATION_PER_VALUE * binCount; + + this.dataValueProperty.value = sortedBins[ this.currentBinIndex ] + this.binWidthProperty.value / 2; + + console.log( 'Stepping sound. totalSoundDuration = ' + this.totalSoundDuration ); + + this.continuousPropertySoundGenerator.step( dt ); + + // If we have completed the sound for the previous bin, adjust the frequency of the sound generator for the next bin + + + // If we have completed the sound for the last bin, stop playing + // if ( this.timeSinceStarted > this.totalSoundDuration ) { + // this.isPlaying = false; + // // Stop the sound + // } + } + } + + // private frequencyForDistance( distance: number ): number { + // // Map the distance to a frequency + // return 400 + distance * 100; + // } +} + +projectileDataLab.register( 'HistogramSoundManager', HistogramSoundManager ); \ No newline at end of file ```

[matthew-blackman]

During the 2/20 sound design @emily-phet and @Ashton-Morris, we discussed ways to simplify the histogram sonification so that it is easier to attribute a sequence of sounds to the shape of a histogram. We agreed to try the following specifications for the sonification of the histogram:

1. Play a single tone for each column, starting with the leftmost column and maintaining a uniform cadence going to the right, pausing if there is an empty bin.
2. For each bin, play a sound with a pitch mapping to the height of the column. To avoid conflicting with the sonification of the field data, we want to try a sound file that does not have a dominant frequency. This will enable us to make the shorter bars sound deeper/more hollow, and the taller bars to sound tinnier.
3. Representing the sonification with a correspond visual is working great.

[samreid]

@matthew-blackman and I collaborated on the implementation, and incorporated feedback from @emily-phet @Ashton-Morris @catherinecarter and @ariel-phet.

The updated sonification plays a single tone for each column with the pitch mapping to the height of the column. The delay between tones is proportional to the width of the bins. We tested under a variety of circumstances and it seems like it is a good direction. Several things could be adjusted such as the exact sound file, the pitch mapping and the duration mapping. We recommend not having "gaps" where there is no histogram data, in favor of a more continuous sequence of tones.

[Ashton-Morris]

I have added some mockup videos as requested to our folder: Mockup Videos

• pdlHistogramRisingPattern.mp4
• pdlHistogramXPattern.mp4
• pdlHistogramMiddleMedianPattern.mp4
• pdlHistogramMockupHighLowMedian.mp4

Also for out records we will be using the sound file pdlCannonLandToneV2.mp3 currently in the histogram.

[matthew-blackman]

The histogram is sounding amazing! We incorporated the recommendations from @emily-phet and @Ashton-Morris and adjusted the pitch range/bin timing so that it has a good balance of playful and meaningful. Fantastic work all! Let's close this issue and continue fine-tuning this feature in https://github.com/phetsims/projectile-data-lab/issues/169.

[catherinecarter]

Reopening to bring up the sonification of a histogram such as this one:

With regards to @samreid's comment above:

We recommend not having "gaps" where there is no histogram data, in favor of a more continuous sequence of tones.

I was expecting to hear the gap between the two pieces of the overall data set. I'm wondering if there would be value to opening the discussion for hearing the gap when there are more than 2 bins (or so) empty to reinforce a larger gap in the data set?

[matthew-blackman]

I'm glad you brought it up @catherinecarter! I think the gaps could work if we use a visual bar that sweeps across the data, and not just highlighting the columns. I wouldn't want too long of a pause without associated visuals in cases like this:

I'll also bring this up in the sound design meeting tomorrow. I think the gaps would help convey the correct proportions of a skew, so I see value here. But I wonder if it would sound strange or unpleasant if there were significant pauses between the sounds. What do you think?

[catherinecarter]

I agree that a significant pause would sound like something is broken. So a short pause would suffice, I suspect, although I'd have to hear it to know for sure. I don't think the size of the gap would need to drive the pause length, so maybe a short pause – long enough to know there's a gap but short enough so it doesn't feel like something is broken – maybe 0.5 seconds or shorter?

I wonder if the current highlight of the bars in the histogram could just be a highlight on the x-axis (the length of the whole gap) at the same cadence as the current sound for adjacent bars would work to achieve the visual pause you mentioned above.

[Ashton-Morris]

I think it sounds particularly satisfying when you hear the same data from different bin widths.

[matthew-blackman]

The remaining work here is being addressed in #174 and #179. Closing.