Unadjusted Hourly Value

[acollad1]

Purpose

• [x] Review REGX pattern for unadjusted hourly value-- does it comply to the JSON schema?
• [x] Verify JSON schema version we are using and whether it supports the look-back operator for scientific notation pattern

Summary of Problem Understanding from Mike Heese's Comment

The user has a problem with the emissions JSON schema. In particular the schema uses a lookbehind assertion in the regular expression to validate the MATS unadjusted hourly value. When this schema is imported into XML Spy it complains that the regular expression is not valid. The JSON Schema Regular Expressions page indicates that the ECMA 262 syntax is used in the Pattern specifications, but the complete syntax is not widely supported.

The assumption is that the user's version of XML Spy does not support the full list of regex syntax in ECMA 262.

Read Research Results Comment

• MATS Significant Digits Requirements
• Suggested RegEx
• Comment on Problem Original RegEx
• Test Values

Conclusion

The lookbehind assertion is not actually needed to create a Regular Expression for Unadjusted Hourly Values, and the existing Unadjusted Hourly Value pattern allows invalid formats. The Unadjusted Hourly Value JSON pattern should use the following Regular Expression:

^((?!0.00)[-]?[0-9][.][0-9]{1,2}[E]([0]|[-]?[1-9][0-9]*)|0.00E0)$

Needed Changes

Update the "emissions.schema.json" in the easey-content repository.

• The repository contains folders for each environment and the changes may need to be migrated through the environments
• For each environment "emissions.schema.json" is in the "/ecmps/reporting-instructions" folder.
• Update the unadjustedHourlyValue Pattern for the following objects to "*`^((?!0.00)[-]?[0-9][.][0-9]{1,2}[E]([0]|[-]?[1-9][0-9])|0.00E0)$`**".
  • MATSDerivedHourlyValue
  • MATSMonitorHourlyValue

Acceptance Tests

1. MATSDerivedHourlyValue JSON Object
   • Ensure that unadjustedHourlyValue matching each of the "valid" value formats in the Research Results Comment's Test Values section are valid.
   • Ensure that unadjustedHourlyValue matching each of the "not allowed" and "invalid" value formats in the Research Results Comment's Test Values section are invalid.
2. MATMonitorHourlyValue JSON Object
   • Ensure that unadjustedHourlyValue matching each of the "valid" value formats in the Research Results Comment's Test Values section are valid.
   • Ensure that unadjustedHourlyValue matching each of the "not allowed" and "invalid" value formats in the Research Results Comment's Test Values section are invalid.

[maheese]

Here's the link to the Request Tracker ticket that spawned this request 73214

[maheese]

The user has a problem with the emissions JSON schema.  In particular the schema uses a lookbehind assertion in the regular expression to validate the MATS unadjusted hourly value. 

"MATSMonitorHourlyValue": {
      "title": "MATSMonitorHourlyValue",
      "type": "object",
      "additionalProperties": true,
      "properties": {
        "parameterCode": {
          "type": "string"
        },
        "unadjustedHourlyValue": {
          "type": [
            "string",
            "null"
          ],
          "minimum": 0,
          "maximum": 30,
          "pattern": "^[+\\-]?(?=\\.\\d|\\d)(?:0|[1-9]\\d*)?(?:\\.\\d+)?(?:(?<=\\d)(?:[eE][+\\-]?\\d+))?$"
        },

When this schema is imported into XML spy it complains that the regular expression is not valid.

The answer to this question is on this page https://json-schema.org/understanding-json-schema/reference/regular_expressions.  Here is the excerpt:

The pattern and patternProperties keywords use regular expressions to express constraints. The regular expression syntax used is from JavaScript (ECMA 262, specifically). However, that complete syntax is not widely supported, therefore it is recommended that you stick to the subset of that syntax described below.

We did not follow this recommendation in defining the schema as the lookbehind assertion is not in the list of "recommended" syntax.  My guess is the regular expression engine which is shipped with this users version of XML Spy does not support the full list of regex syntax in ECMA 262 that's why they get the error.

[djw4erg]

MATS Significant Digits Requirements

1. Must include both a base number and an exponent,
2. The base number must have one integer digit, and one or two decimal digits.
3. The exponent must be preceded by an "E".
4. Both the base number and exponent can begin with a negative sign.
5. When the base number is 0.00 then the exponent must be 0.
6. No spaces nor whitespace are allowed.

Suggested RegEx

^((?!0.00)[-]?[0-9][.][0-9]{1,2}[E]([0]|[-]?[1-9][0-9]*)|0.00E0)$

Comment on original RegEx

^[+\\-]?(?=\\.\\d|\\d)(?:0|[1-9]\\d*)?(?:\\.\\d+)?(?:(?<=\\d)(?:[eE][+\\-]?\\d+))?$

1. It appears to contain escapes for each backslash likely for JSON purposes, hence the number of "\".
2. The implementation of this version of the Unadjusted Hourly Value pattern allows values that are not valid.
3. Based on the Suggested RegEx above lookbehind is not needed.

Test Values

Valid Values

0.00E0 8.58E-2 -8.58E-2 -8.58E2 8.58E2 8.5E-2 -8.5E-2 8.5E2 -8.5E2 8.5E-20 8.58E2 0.00E0

Plus Signs, Lowercase "e", and Spaces Not Allowed

8.58e2 +8.58E2 8.58E+2 +8.58E2 +8.58E+2 8.58 E2

Other Invalid Values

0.00E2 8.5E-0 8.5E-00 8.5E-01 858E-2 85.8E-2 -8.580E-2 +8.580E-2 .858E-2 8.5.8E-2

[djw4erg]

Suggested Next Steps

1. Discuss the results from the MATS Significant Digits Requirements comment with @maheese and @ibarra-michelle to make sure the they agree with the next step.
2. Update the "emissions.schema.json" in the easey-content repository.
   • The repository contains folders for each environment and the changes may need to be migrated through the environments
   • For each environment "emissions.schema.json" is in the "/ecmps/reporting-instructions" folder.
   • Update the unadjustedHourlyValue Pattern for the following objects to "*^((?!0.00)[-]?[0-9][.][0-9]{1,2}[E]([0]|[-]?[1-9][0-9])|0.00E0)$**".
     • MATSDerivedHourlyValue
     • MATSMonitorHourlyValue

[lgiannini1]

Verified on dev using ORIS 628, 4

• Import errors fire when invalid unadjustedHourlyValues are present in matsMonitorHourlyValueData and matsDerivedHourlyValueData records (Example file: Emissions _ Export - Crystal River (4) - 2023 - Q3.json)
• File imports successfully when valid unadjustedHourlyValues are present in matsMonitorHourlyValueData and matsDerivedHourlyValueData records (Example file: Emissions _ Export - Crystal River (4) - 2023 - Q3 (1).json)

[ntknguyen]

The issue is verified.