phetsims / axon

Axon provides powerful and concise models for interactive simulations, based on observable Properties and related patterns.
MIT License
11 stars 8 forks source link

Create a Proxy-based Array implementation that has full API compatibility #330

Closed samreid closed 3 years ago

samreid commented 3 years ago

From https://github.com/phetsims/axon/issues/311, we would like to create a Proxy-based Array implementation that has full API compatibility, which will make it easier to migrate from plain arrays to arrays that can be observed. We discussed that this solution will likely have these disadvantages:

  1. lower performance
  2. more difficulty in code navigation (e.g. navigating to an AxonArrayProxy method)
  3. difficulty extending AxonArrayProxy, like BunnyArray extends AxonArrayProxy

We agreed it will be OK to not be able to subclass this solution (difficulty in extending Proxy discussed in https://github.com/phetsims/axon/issues/311#issuecomment-682882281), but that we will need to characterize performance to see if it is acceptable. Our main concern is that if one Array Proxy is passed to performance-critical code, it could deoptimize all of its calls.

samreid commented 3 years ago

The Proxy-based solution is working well, I think it has compatibility with Array API. I also added adapters to give compatibility with ObservableArray API to aid in migration. I took it for a test drive in Natural Selection, and it was straightforward to convert ObservableArray => createArrayProxy. Even though this doesn't use subtyping, there is a way to simulate BunnyArray (not as nice as subclassing, but it seems workable).

```diff Index: js/common/model/PopulationModel.js IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- js/common/model/PopulationModel.js (revision eef23203cb7af9cf7931b75a1e16cf915abd545b) +++ js/common/model/PopulationModel.js (date 1600558272085) @@ -7,9 +7,9 @@ */ import BooleanProperty from '../../../../axon/js/BooleanProperty.js'; +import createArrayProxy from '../../../../axon/js/createArrayProxy.js'; import DerivedProperty from '../../../../axon/js/DerivedProperty.js'; import NumberProperty from '../../../../axon/js/NumberProperty.js'; -import ObservableArray from '../../../../axon/js/ObservableArray.js'; import Property from '../../../../axon/js/Property.js'; import PropertyIO from '../../../../axon/js/PropertyIO.js'; import Range from '../../../../dot/js/Range.js'; @@ -87,39 +87,39 @@ // @public {ObservableArray.} - data points, for total population and the population of each allele. // Vector2.x = generation, Vector2.y = population - this.totalPoints = new ObservableArray( { + this.totalPoints = createArrayProxy( { + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), tandem: dataPointsTandem.createTandem( 'totalPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), phetioDocumentation: 'Population data points for all bunnies' } ); - this.whiteFurPoints = new ObservableArray( { + this.whiteFurPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'whiteFurPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for bunnies with white fur' } ); - this.brownFurPoints = new ObservableArray( { + this.brownFurPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'brownFurPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), - phetioDocumentation: 'Population data points for bunnies with brown fur' + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), + phetioDocumentation: 'Population data points for bunnies with brown fur', } ); - this.straightEarsPoints = new ObservableArray( { + this.straightEarsPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'straightEarsPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for bunnies with straight ears' } ); - this.floppyEarsPoints = new ObservableArray( { + this.floppyEarsPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'floppyEarsPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for bunnies with floppy ears' } ); - this.shortTeethPoints = new ObservableArray( { + this.shortTeethPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'shortTeethPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for bunnies with short teeth' } ); - this.longTeethPoints = new ObservableArray( { + this.longTeethPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'longTeethPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for bunnies with long teeth' } ); @@ -281,7 +281,7 @@ * @param {number} count */ function recordCount( observableArray, timeInGenerations, count ) { - assert && assert( observableArray instanceof ObservableArray, 'invalid observableArray' ); + assert && assert( Array.isArray( observableArray ), 'invalid observableArray' ); assert && assert( NaturalSelectionUtils.isNonNegative( timeInGenerations ), 'invalid generation' ); assert && assert( NaturalSelectionUtils.isNonNegativeInteger( count ), 'invalid count' ); Index: js/common/model/BunnyArray.js =================================================================== --- js/common/model/BunnyArray.js (revision eef23203cb7af9cf7931b75a1e16cf915abd545b) +++ js/common/model/createBunnyArray.js (date 1600558327840) @@ -6,7 +6,7 @@ * @author Chris Malley (PixelZoom, Inc.) */ -import ObservableArray from '../../../../axon/js/ObservableArray.js'; +import createArrayProxy from '../../../../axon/js/createArrayProxy.js'; import Property from '../../../../axon/js/Property.js'; import PropertyIO from '../../../../axon/js/PropertyIO.js'; import merge from '../../../../phet-core/js/merge.js'; @@ -16,52 +16,42 @@ import Bunny from './Bunny.js'; import BunnyCounts from './BunnyCounts.js'; -class BunnyArray extends ObservableArray { - - /** - * @param {Object} [options] - */ - constructor( options ) { - - options = merge( { +const createBunnyArray = options => { + options = merge( { - // phet-io - tandem: Tandem.REQUIRED, - phetioType: ObservableArray.ObservableArrayIO( ReferenceIO( Bunny.BunnyIO ) ), - phetioState: false - }, options ); - - super( options ); + phetioType: createArrayProxy.ArrayProxyIO( ReferenceIO( Bunny.BunnyIO ) ), + + // phet-io + tandem: Tandem.REQUIRED, + phetioState: false + }, options ); + const bunnyArray = createArrayProxy( options ); - // @public (read-only) {Property.} - this.countsProperty = new Property( BunnyCounts.withZero(), { - tandem: options.tandem.createTandem( 'countsProperty' ), - phetioType: PropertyIO( BunnyCounts.BunnyCountsIO ), - phetioState: false // because counts will be restored as Bunny instances are restored to BunnyGroup - } ); + // @public (read-only) {Property.} + bunnyArray.countsProperty = new Property( BunnyCounts.withZero(), { + tandem: options.tandem.createTandem( 'countsProperty' ), + phetioType: PropertyIO( BunnyCounts.BunnyCountsIO ), + phetioState: false // because counts will be restored as Bunny instances are restored to BunnyGroup + } ); - // Update counts when a bunny is added. removeItemAddedListener is not necessary. - this.addItemAddedListener( bunny => { - this.countsProperty.value = this.countsProperty.value.plus( bunny ); - assert && assert( this.countsProperty.value.totalCount === this.length, 'counts out of sync' ); - } ); + // Update counts when a bunny is added. removeItemAddedListener is not necessary. + bunnyArray.addItemAddedListener( bunny => { + bunnyArray.countsProperty.value = bunnyArray.countsProperty.value.plus( bunny ); + assert && assert( bunnyArray.countsProperty.value.totalCount === bunnyArray.length, 'counts out of sync' ); + } ); - // Update counts when a bunny is removed. removeItemAddedListener is not necessary. - this.addItemRemovedListener( bunny => { - this.countsProperty.value = this.countsProperty.value.minus( bunny ); - assert && assert( this.countsProperty.value.totalCount === this.length, 'counts out of sync' ); - } ); - } + // Update counts when a bunny is removed. removeItemAddedListener is not necessary. + bunnyArray.addItemRemovedListener( bunny => { + bunnyArray.countsProperty.value = bunnyArray.countsProperty.value.minus( bunny ); + assert && assert( bunnyArray.countsProperty.value.totalCount === bunnyArray.length, 'counts out of sync' ); + } ); - /** - * @public - * @override - */ - dispose() { - assert && assert( false, 'dispose is not supported, exists for the lifetime of the sim' ); - super.dispose(); - } -} + bunnyArray.dispose = () => { + assert && assert( false, 'dispose not supported, exists for the life of the sim' ); + }; -naturalSelection.register( 'BunnyArray', BunnyArray ); -export default BunnyArray; \ No newline at end of file + return bunnyArray; +}; + +naturalSelection.register( 'createBunnyArray', createBunnyArray ); +export default createBunnyArray; \ No newline at end of file Index: js/common/model/BunnyCollection.js IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- js/common/model/BunnyCollection.js (revision eef23203cb7af9cf7931b75a1e16cf915abd545b) +++ js/common/model/BunnyCollection.js (date 1600555796008) @@ -20,7 +20,7 @@ import NaturalSelectionQueryParameters from '../NaturalSelectionQueryParameters.js'; import NaturalSelectionUtils from '../NaturalSelectionUtils.js'; import Bunny from './Bunny.js'; -import BunnyArray from './BunnyArray.js'; +import createBunnyArray from './createBunnyArray.js'; import BunnyGroup from './BunnyGroup.js'; import EnvironmentModelViewTransform from './EnvironmentModelViewTransform.js'; import GenePool from './GenePool.js'; @@ -64,12 +64,12 @@ }, options ); // @public (read-only) the live bunnies in bunnyGroup - this.liveBunnies = new BunnyArray( { + this.liveBunnies = createBunnyArray( { tandem: options.tandem.createTandem( 'liveBunnies' ) } ); // @private the dead bunnies in bunnyGroup - this.deadBunnies = new BunnyArray( { + this.deadBunnies = createBunnyArray( { tandem: options.tandem.createTandem( 'deadBunnies' ) } ); @@ -77,7 +77,7 @@ // soon as possible. Mutants are added to this array when born, and removed as soon as they have mated with // another bunny that has the same mutant allele. See also the 'Recessive Mutants' section of model.md at // https://github.com/phetsims/natural-selection/blob/master/doc/model.md#recessive-mutants. - this.recessiveMutants = new BunnyArray( { + this.recessiveMutants = createBunnyArray( { tandem: options.tandem.createTandem( 'recessiveMutants' ), phetioDocumentation: 'for internal PhET use only' } ); ```
samreid commented 3 years ago

To try to understand the performance implications of the Proxy based approach, I used it in a central point in scenery, like so:

I changed Node.js from:

this._children = []; // {Array.<Node>} - Ordered array of child Nodes.

to

this._children = createArrayProxy(); // {Array.<Node>} - Ordered array of child Nodes.

Note: this is not meant to be a realistic simulation of what it would be like if createArrayProxy was used in a few places in simulation specific code, but rather as a "worst case scenario". I ran natural selection with ?secondsPerGeneration=1 and pressed "Add a Mate". On the left is [] and on the right is createArrayProxy, which does all of the element emitter work in addition to the overhead of Proxy.

image

For instance, with [], getChildren takes 3.1 seconds of self-time and with createArrayProxy it takes 7.6% of self-time.

Subjectively, performance in natural selection looked smooth in both cases. I also measured that there are 1644 of those arrayProxy allocations in this test, and that number didn't increase over bunny generations.

EDIT: Also this performance test used the Natural Selection patch from the preceding comment (Natural Selection uses createArrayProxy instead of ObservableArray).

samreid commented 3 years ago

I also batch converted all new ObservableArray to createArrayProxy and that helped catch a few bugs, fixed above. Many sims are passing aqua fuzzing. Also, I was interested to see that when making that swap, all of the ObservableArray unit tests passed (using createArrayProxy instead of ObservableArray). I noticed that the reduce arguments are swapped--that will need to be rectified.

samreid commented 3 years ago

I wrote to @jonathanolson on slack:

Can we schedule a time to discuss createArrayProxy.js? It is working well, but I’d like to get your recommendations on how to improve its performance and how to run pperformance testing.

Also a reminder not to make common code changes until EFAC is published, since master and a branch are being maintained in parallel.

Some notes about potential performance improvements:

samreid commented 3 years ago

@jonathanolson and I ran a deoptimization in Sim.js like so:


const a = createArrayProxy();
a.push( 'hello' );
a.push( 'hello' );
a.push( 'hello' );

_.map( a, t => t );
_.indexOf( a, 'bye' );
_.indexOf( a, 'hello' );
_.uniq( a );
_.some( a, t => t === 'hello' );
_.includes( a, 'hello' );
const b = new Node();

const childArray = createArrayProxy();
const childArray2 = createArrayProxy();
childArray.push( new Text( 'hello' ) );
childArray2.push( new Text( 'bye' ) );
b.children = childArray;
b.accessibleOrder = childArray2;

Then we compared using the snapshot-comparison tool with 100 frames (instead of 10).

Firefox: image

Chrome: image

Safari: image

samreid commented 3 years ago

On Chrome, replacing every ObservableArray with createArrayProxy:

image

For all these tests, createSnapshot has:

    for ( let i = 0; i < 100; i++ ) {
      sendFuzz( 100 );
      sendStep( random.nextDouble() * 0.5 + 0.016 );
    }

Tests from the previous comment had this in BASEModel:

    for (let i=0;i<10000;i++){
      _.indexOf(myArray,'hello');
    }

But this test didn't have it.

samreid commented 3 years ago

Summarizing results from discussion with @jonathanolson:

  1. The performance seems a little slower on Firefox (86 seconds to run instead of 80 seconds to run), but there are no major deoptimizations noted. We are pleasantly surprised that performance is mostly the same.
  2. We may be able to improve this performance even further by having createArrayProxy not return a new function each invocation.

Other features to make sure we handle:

We discussed making sure we have robust support for cases like: myArray.push = 'hello'; but we couldn't think of a performance-optimized way to do this, so we would likely say we do not cover this case.

For my notes: the test harness is http://localhost/main/aqua/html/snapshot-comparison.html?sims=acid-base-solutions,gravity-and-orbits,natural-selection,balloons-and-static-electricity

samreid commented 3 years ago

I ran another before/after test on Firefox, and the results were much closer together (this test also does not have

    for (let i=0;i<10000;i++){
      _.indexOf(myArray,'hello');
    }

in BaseModel.js

image

samreid commented 3 years ago

To check reproducibility of the results before I start performance improvements for createArrayProxy, I tested with and without the deoptimizations in Sim.js in Firefox 3x each:

Firefox no deoptimizations
82336 82435 82026

Firefox with deoptimizations 83526 82385 84953

samreid commented 3 years ago

I tried moving pop to a new methods object like so:

Index: js/createArrayProxy.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- js/createArrayProxy.js  (revision 7e1725fe33b989e88dd6ea36dd32e8a47668fa1e)
+++ js/createArrayProxy.js  (date 1600795064717)
@@ -62,6 +62,7 @@
     tandem: options.tandem.createTandem( 'elementRemovedEmitter' ),
     parameters: [ emitterParameterOptions ]
   } );
+  elementRemovedEmitter.addListener( e=>console.log('removed '+e));

   // @public (read-only) observe this, but don't set it. Updated when Array modifiers are called (except array.length=...)
   const lengthProperty = new NumberProperty( 0, {
@@ -72,8 +73,24 @@

   const targetArray = [];

+  const methods = {
+    pop() {
+      const initialLength = targetArray.length;
+      this.lengthProperty.value = this.length;
+      // debugger;
+      const returnValue = Array.prototype.pop.call( this );
+      initialLength > 0 && this.elementRemovedEmitter.emit( returnValue );
+      return returnValue;
+    }
+  };
+
   const arrayProxy = new Proxy( targetArray, {
     get: function( target, key, receiver ) {
+
+      if ( methods.hasOwnProperty( key ) ) {
+        return methods[ key ];
+      }
+
       const value = target[ key ];
       if ( typeof value !== 'function' ) {
         return value;

But it has the problem that calling Array.prototype.pop triggers other notifications so it signifies that the same element is removed twice.

jonathanolson commented 3 years ago

I'm interested to know why that's happening or dig in sometime!

samreid commented 3 years ago

On the left is master, on the right is all ObservableArray ported to createArrayProxy, with the methods factored out, in firefox.

image

samreid commented 3 years ago

I added tests for the cases described in https://github.com/phetsims/axon/issues/330#issuecomment-696837306, and it is all working OK. I also swapped the order of the parameters in ObservableArray.reduce to make a migration easier. I also addressed a bad phetioType that appeared in charges-and-fields, to make the migration easier.

@jonathanolson can you please review createArrayProxy and its unit tests? I'm interested in determining whether we can replace all occurrences of AxonArray and ObservableArray with this, and eliminate them.

samreid commented 3 years ago

Let's touch base at developer meeting:

Summary of Status:

Questions and Next Steps

Denz1994 commented 3 years ago

From Dev Meeting on 09/24/20:

@pixelzoom: The timeline for working on this issue should not conflict with Natural Selection 1.2.0. @zepumph mentioned that momentum on this issue is valuable and delaying may slow done further work on this issue.

What other review or investigation should be done before production usage?

@samreid mentioned that @pixelzoom should test drive these changes (~30 minutes). It should also be noted that the phet-io api for ObservableArrayIO has changed and should be reviewed.

@pixelzoom is cleared to work test drive with some help from @samreid.

How to document in JSDoc that a parameter is an object created by createArrayProxy? Do we need a typedef?

@pixelzoom will make a recommendation during the test drive.

Denz1994 commented 3 years ago

From Dev Meeting on 09/24/20:

Can we replace all usages of ObservableArray with this? Can ObservableArray be deleted? Can we replace all usages of AxonArray with this? Can AxonArray be deleted?

Devs are signing off on deleting ObservableArray and AxonArray.

zepumph commented 3 years ago

For jsDoc, I think the best way forward would be to create something like ArrayProxyDef.js, and mark that in jsdoc everywhere that the return value of createArrayProxy is used.

samreid commented 3 years ago

Self-assigning to write up review instructions, I'll do this in the next day or two.

samreid commented 3 years ago

Some steps you could take as part of the review:

Take it for a test drive in Natural Selection

  1. Run grunt generate-phet-io-api in natural selection first, to get an initial snapshot of the PhET-iO API

The following steps 2-5 are in this patch:

```diff Index: js/common/model/BunnyArray.js =================================================================== --- js/common/model/BunnyArray.js (revision 7955592612015a808a8e0954abd006c87c5ae566) +++ js/common/model/BunnyArray.js (revision 7955592612015a808a8e0954abd006c87c5ae566) @@ -1,67 +0,0 @@ -// Copyright 2020, University of Colorado Boulder - -/** - * BunnyArray is an ObservableArray of Bunny instances, with counts for each phenotype. - * - * @author Chris Malley (PixelZoom, Inc.) - */ - -import ObservableArray from '../../../../axon/js/ObservableArray.js'; -import Property from '../../../../axon/js/Property.js'; -import PropertyIO from '../../../../axon/js/PropertyIO.js'; -import merge from '../../../../phet-core/js/merge.js'; -import Tandem from '../../../../tandem/js/Tandem.js'; -import ReferenceIO from '../../../../tandem/js/types/ReferenceIO.js'; -import naturalSelection from '../../naturalSelection.js'; -import Bunny from './Bunny.js'; -import BunnyCounts from './BunnyCounts.js'; - -class BunnyArray extends ObservableArray { - - /** - * @param {Object} [options] - */ - constructor( options ) { - - options = merge( { - - // phet-io - tandem: Tandem.REQUIRED, - phetioType: ObservableArray.ObservableArrayIO( ReferenceIO( Bunny.BunnyIO ) ), - phetioState: false - }, options ); - - super( options ); - - // @public (read-only) {Property.} - this.countsProperty = new Property( BunnyCounts.withZero(), { - tandem: options.tandem.createTandem( 'countsProperty' ), - phetioType: PropertyIO( BunnyCounts.BunnyCountsIO ), - phetioState: false // because counts will be restored as Bunny instances are restored to BunnyGroup - } ); - - // Update counts when a bunny is added. removeItemAddedListener is not necessary. - this.addItemAddedListener( bunny => { - this.countsProperty.value = this.countsProperty.value.plus( bunny ); - assert && assert( this.countsProperty.value.totalCount === this.length, 'counts out of sync' ); - } ); - - // Update counts when a bunny is removed. removeItemAddedListener is not necessary. - this.addItemRemovedListener( bunny => { - this.countsProperty.value = this.countsProperty.value.minus( bunny ); - assert && assert( this.countsProperty.value.totalCount === this.length, 'counts out of sync' ); - } ); - } - - /** - * @public - * @override - */ - dispose() { - assert && assert( false, 'dispose is not supported, exists for the lifetime of the sim' ); - super.dispose(); - } -} - -naturalSelection.register( 'BunnyArray', BunnyArray ); -export default BunnyArray; \ No newline at end of file Index: js/common/model/ProportionsModel.js IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- js/common/model/ProportionsModel.js (revision 7955592612015a808a8e0954abd006c87c5ae566) +++ js/common/model/ProportionsModel.js (date 1600981705292) @@ -7,6 +7,7 @@ */ import BooleanProperty from '../../../../axon/js/BooleanProperty.js'; +import createArrayProxy from '../../../../axon/js/createArrayProxy.js'; import DerivedProperty from '../../../../axon/js/DerivedProperty.js'; import NumberProperty from '../../../../axon/js/NumberProperty.js'; import ObservableArray from '../../../../axon/js/ObservableArray.js'; @@ -94,9 +95,9 @@ phetioDocumentation: 'Counts at the start of the current generation' } ); - const previousCounts = new ObservableArray( { + const previousCounts = createArrayProxy( { tandem: options.tandem.createTandem( 'previousCounts' ), - phetioType: ObservableArray.ObservableArrayIO( ProportionsCounts.ProportionsCountsIO ), + phetioType: createArrayProxy.ArrayProxyIO( ProportionsCounts.ProportionsCountsIO ), phetioDocumentation: 'Start and End counts for previous generations, indexed by generation number' } ); Index: js/common/model/PopulationModel.js IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- js/common/model/PopulationModel.js (revision 7955592612015a808a8e0954abd006c87c5ae566) +++ js/common/model/PopulationModel.js (date 1600982861089) @@ -7,6 +7,7 @@ */ import BooleanProperty from '../../../../axon/js/BooleanProperty.js'; +import createArrayProxy from '../../../../axon/js/createArrayProxy.js'; import DerivedProperty from '../../../../axon/js/DerivedProperty.js'; import NumberProperty from '../../../../axon/js/NumberProperty.js'; import ObservableArray from '../../../../axon/js/ObservableArray.js'; @@ -87,39 +88,39 @@ // @public {ObservableArray.} - data points, for total population and the population of each allele. // Vector2.x = generation, Vector2.y = population - this.totalPoints = new ObservableArray( { + this.totalPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'totalPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for all bunnies' } ); - this.whiteFurPoints = new ObservableArray( { + this.whiteFurPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'whiteFurPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for bunnies with white fur' } ); - this.brownFurPoints = new ObservableArray( { + this.brownFurPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'brownFurPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for bunnies with brown fur' } ); - this.straightEarsPoints = new ObservableArray( { + this.straightEarsPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'straightEarsPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for bunnies with straight ears' } ); - this.floppyEarsPoints = new ObservableArray( { + this.floppyEarsPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'floppyEarsPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for bunnies with floppy ears' } ); - this.shortTeethPoints = new ObservableArray( { + this.shortTeethPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'shortTeethPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for bunnies with short teeth' } ); - this.longTeethPoints = new ObservableArray( { + this.longTeethPoints = createArrayProxy( { tandem: dataPointsTandem.createTandem( 'longTeethPoints' ), - phetioType: ObservableArray.ObservableArrayIO( Vector2IO ), + phetioType: createArrayProxy.ArrayProxyIO( Vector2IO ), phetioDocumentation: 'Population data points for bunnies with long teeth' } ); @@ -276,17 +277,17 @@ /** * Records a count if it differs from the previous data point. - * @param {ObservableArray.} observableArray + * @param {Vector2[]} array * @param {number} timeInGenerations - time (in generations) for the count * @param {number} count */ -function recordCount( observableArray, timeInGenerations, count ) { - assert && assert( observableArray instanceof ObservableArray, 'invalid observableArray' ); +function recordCount( array, timeInGenerations, count ) { + assert && assert( Array.isArray( array ), 'invalid array' ); assert && assert( NaturalSelectionUtils.isNonNegative( timeInGenerations ), 'invalid generation' ); assert && assert( NaturalSelectionUtils.isNonNegativeInteger( count ), 'invalid count' ); - if ( observableArray.length === 0 || observableArray.get( observableArray.length - 1 ).y !== count ) { - observableArray.push( new Vector2( timeInGenerations, count ) ); + if ( array.length === 0 || array.get( array.length - 1 ).y !== count ) { + array.push( new Vector2( timeInGenerations, count ) ); } } Index: js/common/model/createBunnyArray.js IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- js/common/model/createBunnyArray.js (date 1600983273088) +++ js/common/model/createBunnyArray.js (date 1600983273088) @@ -0,0 +1,56 @@ +// Copyright 2020, University of Colorado Boulder + +/** + * BunnyArray is an ObservableArray of Bunny instances, with counts for each phenotype. + * + * @author Chris Malley (PixelZoom, Inc.) + */ + +import createArrayProxy from '../../../../axon/js/createArrayProxy.js'; +import Property from '../../../../axon/js/Property.js'; +import merge from '../../../../phet-core/js/merge.js'; +import Tandem from '../../../../tandem/js/Tandem.js'; +import ReferenceIO from '../../../../tandem/js/types/ReferenceIO.js'; +import naturalSelection from '../../naturalSelection.js'; +import Bunny from './Bunny.js'; +import BunnyCounts from './BunnyCounts.js'; + +const createBunnyArray = options => { + + options = merge( { + + // phet-io + tandem: Tandem.REQUIRED, + phetioType: createArrayProxy.ArrayProxyIO( ReferenceIO( Bunny.BunnyIO ) ), + phetioState: false + }, options ); + + const bunnyArray = createArrayProxy( options ); + + // @public (read-only) {Property.} + bunnyArray.countsProperty = new Property( BunnyCounts.withZero(), { + tandem: options.tandem.createTandem( 'countsProperty' ), + phetioType: createArrayProxy.ArrayProxyIO( BunnyCounts.BunnyCountsIO ), + phetioState: false // because counts will be restored as Bunny instances are restored to BunnyGroup + } ); + + // Update counts when a bunny is added. removeItemAddedListener is not necessary. + bunnyArray.addItemAddedListener( bunny => { + bunnyArray.countsProperty.value = bunnyArray.countsProperty.value.plus( bunny ); + assert && assert( bunnyArray.countsProperty.value.totalCount === bunnyArray.length, 'counts out of sync' ); + } ); + + // Update counts when a bunny is removed. removeItemAddedListener is not necessary. + bunnyArray.addItemRemovedListener( bunny => { + bunnyArray.countsProperty.value = bunnyArray.countsProperty.value.minus( bunny ); + assert && assert( bunnyArray.countsProperty.value.totalCount === bunnyArray.length, 'counts out of sync' ); + } ); + + bunnyArray.dispose = () => { + assert && assert( false, 'dispose is not supported, exists for the lifetime of the sim' ); + }; + return bunnyArray; +}; + +naturalSelection.register( 'createBunnyArray', createBunnyArray ); +export default createBunnyArray; \ No newline at end of file Index: js/common/model/BunnyCollection.js IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== --- js/common/model/BunnyCollection.js (revision 7955592612015a808a8e0954abd006c87c5ae566) +++ js/common/model/BunnyCollection.js (date 1600982067449) @@ -20,8 +20,8 @@ import NaturalSelectionQueryParameters from '../NaturalSelectionQueryParameters.js'; import NaturalSelectionUtils from '../NaturalSelectionUtils.js'; import Bunny from './Bunny.js'; -import BunnyArray from './BunnyArray.js'; import BunnyGroup from './BunnyGroup.js'; +import createBunnyArray from './createBunnyArray.js'; import EnvironmentModelViewTransform from './EnvironmentModelViewTransform.js'; import GenePool from './GenePool.js'; import PunnettSquare from './PunnettSquare.js'; @@ -64,12 +64,12 @@ }, options ); // @public (read-only) the live bunnies in bunnyGroup - this.liveBunnies = new BunnyArray( { + this.liveBunnies = createBunnyArray( { tandem: options.tandem.createTandem( 'liveBunnies' ) } ); // @private the dead bunnies in bunnyGroup - this.deadBunnies = new BunnyArray( { + this.deadBunnies = createBunnyArray( { tandem: options.tandem.createTandem( 'deadBunnies' ) } ); @@ -77,7 +77,7 @@ // soon as possible. Mutants are added to this array when born, and removed as soon as they have mated with // another bunny that has the same mutant allele. See also the 'Recessive Mutants' section of model.md at // https://github.com/phetsims/natural-selection/blob/master/doc/model.md#recessive-mutants. - this.recessiveMutants = new BunnyArray( { + this.recessiveMutants = createBunnyArray( { tandem: options.tandem.createTandem( 'recessiveMutants' ), phetioDocumentation: 'for internal PhET use only' } ); ```

Here's a copy of createBunnyArray by itself in case the patch doesn't apply:

```js // Copyright 2020, University of Colorado Boulder /** * BunnyArray is an ObservableArray of Bunny instances, with counts for each phenotype. * * @author Chris Malley (PixelZoom, Inc.) */ import createArrayProxy from '../../../../axon/js/createArrayProxy.js'; import Property from '../../../../axon/js/Property.js'; import merge from '../../../../phet-core/js/merge.js'; import Tandem from '../../../../tandem/js/Tandem.js'; import ReferenceIO from '../../../../tandem/js/types/ReferenceIO.js'; import naturalSelection from '../../naturalSelection.js'; import Bunny from './Bunny.js'; import BunnyCounts from './BunnyCounts.js'; const createBunnyArray = options => { options = merge( { // phet-io tandem: Tandem.REQUIRED, phetioType: createArrayProxy.ArrayProxyIO( ReferenceIO( Bunny.BunnyIO ) ), phetioState: false }, options ); const bunnyArray = createArrayProxy( options ); // @public (read-only) {Property.} bunnyArray.countsProperty = new Property( BunnyCounts.withZero(), { tandem: options.tandem.createTandem( 'countsProperty' ), phetioType: createArrayProxy.ArrayProxyIO( BunnyCounts.BunnyCountsIO ), phetioState: false // because counts will be restored as Bunny instances are restored to BunnyGroup } ); // Update counts when a bunny is added. removeItemAddedListener is not necessary. bunnyArray.addItemAddedListener( bunny => { bunnyArray.countsProperty.value = bunnyArray.countsProperty.value.plus( bunny ); assert && assert( bunnyArray.countsProperty.value.totalCount === bunnyArray.length, 'counts out of sync' ); } ); // Update counts when a bunny is removed. removeItemAddedListener is not necessary. bunnyArray.addItemRemovedListener( bunny => { bunnyArray.countsProperty.value = bunnyArray.countsProperty.value.minus( bunny ); assert && assert( bunnyArray.countsProperty.value.totalCount === bunnyArray.length, 'counts out of sync' ); } ); bunnyArray.dispose = () => { assert && assert( false, 'dispose is not supported, exists for the lifetime of the sim' ); }; return bunnyArray; }; naturalSelection.register( 'createBunnyArray', createBunnyArray ); export default createBunnyArray; ```
  1. Replace occurrences of new ObservableArray with createArrayProxy, also change the imports.
  2. In the changed files, replace phetioType: ObservableArray.ObservableArrayIO with phetioType: createArrayProxy.ArrayProxyIO
  3. Change BunnyArray.js to createBunnyArray.js (this part is kind of sketchy, and not very amenable to further subtyping)
  4. Change the check for instanceof ObservableArray
  5. Run grunt generate-phet-io-api in natural selection, for comparison. Note the added elementAddedEmitters, etc. and that there are no more ObservableArray. Note that the tandem structure is the same (aside from the new items).
  6. Test the sim by itself, test in studio, test the state wrapper.

Review the implementation in createArrayProxy:

  1. You may wish to review https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Proxy for background information
  2. Review createArrayProxy.js, with an eye for API, problematic implementation details, etc.
  3. Review and test the unit tests in axon/js/createArrayProxyTests.js

Questions

  1. Are any API/implementation/documentation changes necessary before we convert all simulations to use createArrayProxy and eliminate ObservableArray and AxonArray?
  2. Any embellishments that could be made after broad adoption?
  3. Would you recommend adding any unit tests?
  4. Note the section of methods that are solely for ObservableArray API compatibility. How eagerly should we try to get rid of these? I'm talking about methods like count, find, shuffle, getArrayCopy, etc. Note the 2 TODOs are in this category.
  5. Comment on whether we can move to this new pattern in master now or if it would be better to wait until Natural Selection has another set of SHAs.
  6. Please recommend how QA should be involved in these changes, if at all.
  7. Can you think of a better name than createArrayProxy and ArrayProxyIO?

UPDATES

I committed ArrayProxyDef as prescribed by @zepumph and now cases like

* @param {ObservableArray.<Vector2>} observableArray

can be replaced by:

* @param {ArrayProxyDef} arrayProxy

and it has nice support for autocomplete, thanks @zepumph!

pixelzoom commented 3 years ago

After a quick code review...

// @public (read-only) (ArrayProxyIO)
AxonArray.ArrayProxyPhetioObject = ArrayProxyPhetioObject;
395 // @public (read-only) (ArrayProxyIO)
23 * @param {arrayProxy[]} arrayProxy
  if ( options && options.hasOwnProperty( 'length' ) ) {
    assert && assert( !options.hasOwnProperty( 'elements' ), 'options.elements and options.length are mutually exclusive' );
  }
288      // console.log( `Changing ${key} (type===${typeof key}), from ${oldValue} to ${newValue}` );

315      // console.log( `deleteProperty ${key}, ${typeof key}` );
        removedElements.forEach( element => elementRemovedEmitter.emit( element ) );
378   * @param {Object} arrayProxy
samreid commented 3 years ago

Thanks, I committed fixes for the recommendations above.

pixelzoom commented 3 years ago

@samreid spent some time with me on Zoom, answering questions, and bringing me up to speed on how Proxy/Reflect are used. Then we reviewed my "test drive" in Natural Selection.

Next step will be to merge my Natural Selection test drive into master, and replace some of the methods that are specific to ObservableArray. Then @samreid will address REVIEW comments and globally replace ObservableArray.

pixelzoom commented 3 years ago

@samreid and I spent a little more time on Zoom, working on issues related to WebStorm type recognition.

In an example like this:

1 /**
2  * @param {Array} array
3  */
4 function doSomething( array ) {...}
5 
6 const arrayProxy = createArrayProxy( ... );
7 
8 doSomething( arrayProxy );

... WebStorm flags the arrayProxy argument on line 8 with "Argument type ArrayProxyDef is not assignable to parameter type Array". So it doesn't know that ArrayProxyDef is an Array.

NS has examples like this in PopulationModel.js (calls to recordCounts) and DataProbe.js (calls to this.getCount). In those cases, the Array element type is specified (e.g. {Array.<Vector2>}) which may be a further complication.

We don't think this is a deal breaker, but it needs more investigation.

pixelzoom commented 3 years ago

createArrayProxy is causing a performance problem in Natural Selection. See https://github.com/phetsims/natural-selection/issues/252#issuecomment-700326821.

samreid commented 3 years ago

In slack, I said:

This seems to work OK, until we get the typedef working:

class ArrayProxyDef extends Array { // eslint-disable-line
  constructor() {
    assert && assert( false, 'This class is for documentation and Webstorm support only and should not be instantiated' );
    super();
    this.lengthProperty = new NumberProperty( 0 );
    this.elementAddedEmitter = new Emitter();
    this.elementRemovedEmitter = new Emitter();
  }
}

@pixelzoom replied:

That looks dangerous (even with the assertion). I don't think we want an entire Array subclass masquerading/paralleling createArrayProxy. I'd rather live with the WebStorm complaints.

pixelzoom commented 3 years ago

Next steps discussed with @samreid on Slack:

Sam Reid 10:53 AM Do you think you will be able to address https://github.com/phetsims/axon/issues/330#issuecomment-700277191 before you leave? Also, that comment makes it sound like I should wait for NS before I take further steps. Is that accurate?

Chris Malley 11:07 AM Assuming that performance in NS has been resolved (see https://github.com/phetsims/natural-selection/issues/252#issuecomment-702270823)... I don't think that https://github.com/phetsims/axon/issues/331 is a deal breaker. Looking at REVIEW comments for createArrayProxy, this one seems like it should be addressed before proceeding with replacement throughout sims:

//REVIEW https://github.com/phetsims/axon/issues/330 wondering if createArrayProxy is the best name for this.
//  It exposes the details of how it's implemented (Proxy) and there could be other Proxy implementations that
//  add other features. It would be better if the name described what features it adds to Array, not how those
//  features are added. I unfortunately can't think of better name than createObservableArray.

Sam Reid 11:10 AM I’m OK with createObservableArray and ObservableArrayDef, I can’t think of something better Sound good to you?

Chris Malley 11:10 AM Yes, I think that's the best option. "observable array" definitely is the best description of what this is.

pixelzoom commented 3 years ago

From https://github.com/phetsims/axon/issues/330#issuecomment-700277191:

Next step will be to merge my Natural Selection test drive into master, and replace some of the methods that are specific to ObservableArray. Then @samreid will address REVIEW comments and globally replace ObservableArray.

Natural Selection (master) has been converted to createObservableArray, see https://github.com/phetsims/natural-selection/issues/252#issuecomment-702350034.

When you rename to createObservableArray and ObservableArrayDef, please hit the occurrences of ArrayProxyDef in NS.

samreid commented 3 years ago

//REVIEW https://github.com/phetsims/axon/issues/330 do we need methods.slice?

I don't think we need to add methods.slice, since slice is already defined on the proxy.

samreid commented 3 years ago

I renamed it to createObservableArray and addressed REVIEW issues. Some I responded with a comment above, some moved to side issues. Ready to review my changes, and I'll create a new issue to migrate sims to use this.

pixelzoom commented 3 years ago

Changes looks good. I don't see anything else that needs to be done here, so closing.