Front End Makes Bad Requests to Back-End: Unaligned Dates

[lepeuvedic-natixar]

Problem: The back-end is specified to reject requests if the dates are not well aligned considering the associated time step. The time step can be specified as an integer value followed by a letter: m for minutes, h for hours, d for days, w for weeks, M for months, Q for quarters, y for years. The only currently supported integer value is 1. Omitting it is equivalent to specifying 1.

The front end currently always requests "scale=1m" and the alignment constraint means that the dates should always be YYYY-MM-01T00:00+00:00

Steps to Reproduce:

1. Set the proxy in debug mode and capture all the requests (won't be necessary when the back-end actually starts rejecting such requests, since the default proxy mode captures all failures.
2. Check "Month" is selected on both Total Emissions charts.
3. Observe that any date range selected is passed verbatim without alignment as a request.
4. Observe that hour, minutes, secondes and time zone are missing, when only seconds are optional (and must be 00 when present).

Desired Behavior: The goal is to make it impossible to generate a chart using partial data. If the database contain data up to May 3rd, 2024, it shall not be possible to display monthly data for the month of May 2024, because the absolute emissions would be small not because of any progress made, but because only partial data has been recorded for the month of May.

The data range selector should query the back-end to read the available date range in the database for the currently displayed scenario(s) (using the /scenario/{scenarioId} endpoint with the client's default scenario URI). In a scenario compare situation (currently not possible in the front-end), the smallest of all the compared scenarios date ranges should be the limit. While this comparison is not yet possible, the relevant function interface in the front-end should allow for lists of scenarios, not just one scenario, even if they are currently only used with a list containing only one scenario.

The user set dates MUST NOT change when the user clicks on time step selection buttons. This is especially important since the two "Total Emissions" charts can have different time steps.

An inappropriate situation can result from either a choice of start date which is too close to end date, a choice of end date that is too close to start date or a choice of time scale that is too large on any of the two charts. Accordingly, guards and input validation must be in place for all four inputs:

• The maximum acceptable start date shall be at one time step (largest of the two) before the closest aligned date (largest timestep) preceding the current end date. E.g. with 1 month time step and end date June 13th, 2022, the latest selectable start date will be May 1st 2022. Any start date without time is supposed to mean at 00:00 and any end date without time is supposed to mean at 00:00 the next morning (or 24:00 the same day). This convention is chosen because March 1st to March 31st will show the entire month of March, which is what most users expect. Selecting the same date for start and end will result if showing one day worth of data.
• The minimum acceptable end date shall be at one time step (largest of the two) after the closest aligned date (largest timestep) following the current start date. E.g. with 1 month time step and start date Nov 27th, 2022, the earliest selectable end date will be Dec 31st 2022.
• The largest selectable time step on any the two charts will be the largest one that can fit at last 1 aligned period between start date and end date. When time step buttons are not active, a hover bubble shall explain clearly to the user what they need to do to be allowed to change to that time step (change the date range).

All the ISO8601 timestamps can be used in the /data/ranges request. The yyyyWww (week number) ISO8601 format is accepted to simplify week-aligned requests.

With the guards in place, the front-end shall properly align the start and end dates it puts on requests to the back-end. In order to simplify processing and to maintain compatibility with ISO8601 timestamp standard, a date without time is invalid in a request to the backend as specified in API Specification - Data Endpoint.

Relevant part of the specification:

• A list of time ranges which, when combined with a given time scale yields more than 10000 distinct time steps is illegal and the request shall return a BadRequest error code with the explanation "Excessive number of data points requested."
• Start and end timestamps must be aligned with the natural boundary of the applicable time scale. For example, if the time scale is a day or more, the time must be T00:00:00+HH:MM where the last part indicates the time zone (most time zones have MM="00"). The time zone is arbitrary, but must be the same for the start and the end timestamp. Aggregating data to Zulu (UTC) time is also valid, as long as the hour is T00:00:00. Since the smallest time scale is minutes, the seconds part can always be omitted and, if present, shall be ":00". ISO 8601 can also directly encode week numbers, which could be convenient to guarantee alignment with week boundaries. The correct format is "YYYY-Www-D+hh:mm", where D is week day number from 1 to 7 and the last part specifies a time zone, and can be replaced by "Z" to indicate UTC.
• Misaligned timestamps will cause the backend to return a BadRequest with the explanation "{start / end} timestamp is not aligned with {time step} natural boundary".

Note: The precision in italics is not present in the actual specification, but the full ISO8601 W notation is supported.

[lepeuvedic]

The problem with unaligned start/end timestamps comes from here: https://github.com/Natixar/natixar-frontend/blob/d713c04a472807427aae8816cc2cd8afb1b80b3c/src/data/store/features/emissions/ranges/EmissionRangesSlice.ts#L257

const selectTimeRangeReducer: CaseReducer<
  EmissionRangeState,
  PayloadAction<TimeRange>
> = (state, action) => {
  state.dataRetrievalParameters.timeRangeOfInterest = action.payload
}

The selectProtocolReducer function must include logic to align on natural boundaries the state.dataRetrievalParameters.timeRangeOfInterest following the specification, based on the smallest currently chosen time step. Note that the backend supports requesting several time ranges at once, especially in order to avoid having to carry out date-aligned aggregation of data in the front end, but the state vector defined in EmissionRangesSlice.ts does not have the capability to store more than one set in allPoints[]. On top of that, the timeStepInSecondsPattern is stored in the definition of overallTimeWindow (the front end still lacks multiscale points storage capability), but the state of the time scale buttons on the two time-dependent charts is not stored, and the data is not refreshed from the back-end when these buttons are used. All this is somewhat problematic because it is not possible to select more than one-week of data at minute scale, and

https://github.com/Natixar/natixar-frontend/blob/d713c04a472807427aae8816cc2cd8afb1b80b3c/src/data/store/features/emissions/ranges/EmissionRangesSlice.ts#L49

In a nutshell the API specification says two things:

• Given size intervals (minute, hour, day, week, month, quarter, year) start and end on their natural boundaries. Currently weeks start on Monday morning, terminate on Sunday evening (might be a user/organization setting in the future). Anything at the scale of day or more is affected by timezone, and by definition a day starts and ends at T00:00:00. This is enforced in the back-end, and the start and end timestamps in a time range must have the same +HH:MM time zone indication.
• The data shown in an interval must cover the entire interval: in other terms, the tool prohibits biased charts showing a decrease when if fact the dates have been used to truncate the data. If a user tries, they won't get the data for the period.

For example these rules imply that, if the start date (or the absolute beginning of recorded data) is a Wednesday, week-scale display can only occur starting on the next Monday. The front should enforce that requested data is always included in the date range set on the date range picker.

[ChrisNatixar]

Fixed with PR #73