phetsims / query-string-machine

Query String Machine is a query string parser that supports type coercion, default values & validation. No dependencies.
MIT License
3 stars 3 forks source link

public "graceful" warnings for Array type don't work #42

Closed zepumph closed 4 years ago

zepumph commented 4 years ago

Over in https://github.com/phetsims/joist/issues/644 there were two suggestions for being more graceful about how we support invalid parameters as it pertains to the production dialog that shows query parameter errors to users:

(2) screens=Macro,Micro fails hard. This is likely a general problem with not being able to display a warning dialog when the elementSchema of a query parameter is violated. (3) screens=1.2 fails hard. There's apparently no verification that values are integers. Unfortunate since '.' used to be the separator for arrays.

@samreid and I are currently brainstorming a couple of ways to tackle this problem. The main issue is that currently, the warnings only apply to parameters that pass the "valid" check in the QP schema, but for some other reason are invalid. Instead, this should be expanded to all values of query parameters in these cases, so that a full list of warnings can be shown.

zepumph commented 4 years ago

We found that public is the schema field that should be used for this, and is supposed to already be handling this. The issue is that it isn't totally supported for arrays, because the elements aren't treated as public. A quick hack to demonstrate the problem is the following:


* Rerun the sim with the same query parameter, and note that you get the invalid QP warning dialog now. 

So the issue is that we aren't forwarding the `public` field into the element schema for array.

@samreid and I investigated and found a solution based on the current implementation of `public`.  In this, we forward the `public` field to the element schema so that it can be graceful and apply warnings. This works at providing the dialog for `?screens=Micro` or `screens=Micro,Macro`. Committed above.

This work does not seem done yet though, as this doesn't cover the case `?screens=1.1`. While working on this we found that the handling for `public` seems very incomplete. We needed to add the below patch in order to successfully get element schemas to respect `isValidValue`. But even so this feels like the wrong way to approach this problem. The basic algorithm for QSM.getForString is (1) validateSchema, (2) parseValues into appropriate types, then (3) validate those values. We found that  much of the warning infrastructure was build into the parsing, in `getValidValue`. So the above commit builds out that logic for parsing arrays' elements. We noticed that the validation step doesn't validate array elements at all:

`        isValidValue: value => Array.isArray( value ) || value === null`.`

<details><summary>patch that adds isValidValue to parsing functions</summary>

```diff
Index: query-string-machine/js/QueryStringMachine.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- query-string-machine/js/QueryStringMachine.js   (revision fce9acaca3dea7caa639454b052843b691b14193)
+++ query-string-machine/js/QueryStringMachine.js   (date 1595443998256)
@@ -589,10 +589,22 @@
      */
     const parseNumber = function( key, schema, string ) {
       const number = Number( string );
-      const ok = !isNaN( number );
+      const ok = !isNaN( number ) && ( schema.isValidValue ? schema.isValidValue( string ) : true );
       return getValidValue( ok, key, ok ? number : string, schema, `value must be a number for key "${key}"` );
     };

+    /**
+     * Parses the value for a type 'number'.
+     * @param {string} key - the query parameter name
+     * @param {Object} schema - schema that describes the query parameter, see QueryStringMachine.get
+     * @param {string|null} string - value from the query parameter string
+     * @returns {number}
+     */
+    const parseString = function( key, schema, string ) {
+      const ok = schema.isValidValue ? schema.isValidValue( string ) : true;
+      return getValidValue( ok, key, ok ? string : string, schema, `value must be a number for key "${key}"` );
+    };
+
     /**
      * Parses the value for a type 'array'.
      * @param {string} key - the query parameter name
@@ -726,7 +738,7 @@
         required: [],
         optional: [ 'defaultValue', 'validValues', 'isValidValue', 'private', 'public' ],
         validateSchema: null, // no type-specific schema validation
-        parse: ( key, schema, string ) => string, // The variable to be parsed is already string, so it is guaranteed to parse as a string.
+        parse: parseString,
         isValidValue: value => value === null || typeof value === 'string'
       },

Index: chipper/js/initialize-globals.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- chipper/js/initialize-globals.js    (revision beaa4fd18398d2842d46edf320089e194720f32c)
+++ chipper/js/initialize-globals.js    (date 1595443998240)
@@ -417,7 +417,8 @@
     screens: {
       type: 'array',
       elementSchema: {
-        type: 'number'
+        type: 'number',
+        isValidValue: Number.isInteger
       },
       defaultValue: null,
       isValidValue: function( value ) {

My recommendation is to not implement the above patch, and to instead build out an additional step in the isValidValue stage of getForString that iterates through element types and checks isValidValue on them.

@pixelzoom's comment over in https://github.com/phetsims/joist/issues/644#issuecomment-662588001 said it best, marking this as blocks publication:

screens is a public query parameter, so it's supposed to fail gracefully -- display a warning dialog and revert to defaults. It does not do that and fails hard for the 2 outstanding cases described in https://github.com/phetsims/joist/issues/644#issue-663325840. So if it were my call, I feel that this is still unacceptably buggy, and therefore still blocks publication. But I'll be happy to defer to a designer.

samreid commented 4 years ago

run a sim with ?screens=43289. Note the hard fail with a console error.

I ran http://localhost/main/ph-scale-basics/ph-scale-basics_en.html?brand=phet&screens=43289 (note there is no ?ea) and it correctly showed the warning dialog:

image

However, ?screens=hello is still a problem.

samreid commented 4 years ago

@zepumph can you comment whether your proposal at the bottom of the issue description includes propagating public to the array element sub-schemas?

zepumph commented 4 years ago

Sorry for the confusion, the "reproducing the bug" only applies if you don't have https://github.com/phetsims/query-string-machine/commit/fce9acaca3dea7caa639454b052843b691b14193 in your local copy.

@zepumph can you comment whether your proposal at the bottom of the issue description includes propagating public to the array element sub-schemas?

No. That was done in https://github.com/phetsims/query-string-machine/commit/fce9acaca3dea7caa639454b052843b691b14193. The patch below is about supporting ?screens=1.1 and that fact that currently elements of arrays don't support isValidValue. The patch tries to apply that feature during the parsing stage, but probably instead we need to add it to the "validation" stage (next to the three conditionals at the bottom of QSM.getForString that have to do with validation.

zepumph commented 4 years ago

After the above commits, although wrongly named in their message, QSM now respects validValues and isValidValue for array elementSchema. I also added tests for isValidValue and validValues in an array mixed into tests that when public, defaultValues should be gracefully accepted and warnings populated.

The one last piece that still feels weird to me is that elements of the array are checked individually, out context of the greater array as a whole, while being parsed into strings/numbers/etc. If elements are not correctly parsable, then an individual warning will be displayed for each element of the array. I am alright with this, but it is a bit weird, so I thought I would mention it.

For example, try running http://localhost:8080/ph-scale/ph-scale_en.html?brand=phet&ea&screens=Hello1,1,Goose1, and notice that this is the dialog:

image

@arouinfar, as my go-to designer, what do you think about this current behavior of multiple warnings in this one case? All cases listed in the first comment of https://github.com/phetsims/joist/issues/644#issue-663325840 (?screens=1,Hi, ?screens=1.1, ?screens=10,1, ?screens=Macro,Micro ), are all working at successfully being graceful and delivering the dialog.

Let me know if you want more explanation.

pixelzoom commented 4 years ago

There should most definitely not be 2 screens query parameters shown in the above dialog. This dialog is supposed to identify that a query parameter is incorrect and show its complete value. It is (intentionally) not intended to identify what is wrong with the value.

zepumph commented 4 years ago

Makes sense to me! That will involve a mediumish workaround or largeish overhaul of how the public parameter and QSM.warnings are populated because it was done without the "recursive" context that parsing a number could be part of a whole that is larger than just that value. I won't be able to work further on this until Wednesday. Anyone else is welcome to take a look.

chrisklus commented 4 years ago

I investigated this a bit but am not sure of the best way to proceed. Self assigning to pair with @zepumph.

zepumph commented 4 years ago

In the above commits, @chrisklus and I refactored QSM's public feature to only work during the "validation" phase of getForString. We boiled down that method's algorithm into 4 steps:

  1. get values from the query string
  2. validate schema
  3. parse values
  4. validation of the value

Before, there was logic in the parsing step which was validating that the values were as expected, this was using getValidValue to support the public parameter. Instead of keeping that there, we altered parsing functions to be more graceful, and give non-parsed values if parsing didn't work (i.e. parseNumber('hello')-> 'hello'). This set us up well to have that incorrect value then caught by the validation.

After that change, it was easy to have all public graceful logic (via getValidValue) to be in the "validation of the value" step instead of sprinkled there and during parsing.

We also caught a few other bugs while working on this, and added tests.

Now when I load http://localhost:8080/ph-scale/ph-scale_en.html?brand=phet&ea&screens=Hello1,1,Goose1 I get this dialog: image

@pixelzoom please review. Are there any other cases we haven't covered for ph-scale or in general?

arouinfar commented 4 years ago

@zepumph the dialog with screens=Hello1,1,Goose1 looks good to me.

zepumph commented 4 years ago

@pixelzoom, while it is fresh in my mind, here is the list of commits to cherry-pick for this issue (in order):

https://github.com/phetsims/query-string-machine/commit/fce9acaca3dea7caa639454b052843b691b14193 https://github.com/phetsims/chipper/commit/8c544f807e913d2520af4d774647da08353205da https://github.com/phetsims/query-string-machine/commit/af6f253315b0ead707f9d917aa905080e958794f https://github.com/phetsims/query-string-machine/commit/e19d636137ecdffd9fb5111c63b1d40658cf714d https://github.com/phetsims/query-string-machine/commit/d6d4771d2c35d1d231cee1d4036aa9a0d5d6422d https://github.com/phetsims/query-string-machine/commit/deb17dd5ca0cd50e0033e475bc655d77dc6eeff3 https://github.com/phetsims/chipper/commit/bd228a8a821dd775e61fd3c8de0a5f7fa21a5562

zepumph commented 4 years ago

Likely applies https://github.com/phetsims/QA/issues/509

pixelzoom commented 4 years ago

There's still a problem here. With ?ea, something like screens=5 (in a 3-screen sim) will throw an assertion. That's not the case with screens=Hello1,1,Goose1. So behavior with ?ea is inconsistent.

pixelzoom commented 4 years ago

@zepumph and I discussed via Zoom. Other than the assertions in selectScreens.js that are required for unit tests, public query parameters generally do not assert out. And making them assert out would require a QSM dependency on assert.js. So we're not going to worry about this, and live with the assertion inconsistency between public and non-public query parameters. If anyone feels differently, please create a separate non-blocking issue.

I'll proceed with cherry picking.

pixelzoom commented 4 years ago

Cherry-pick of https://github.com/phetsims/query-string-machine/issues/42#issuecomment-665983384 completed.

Closing.