Maintain history when a monitor is modified and include sequence of steps

[paulb-elastic]

Monitors can often change over time as the website under test may subtly change.

For example, imagine a monitor with a journey configured with the following steps:

• My Monitor
  • Step 1 - Homepage
  • Step 2 - Click My Account
  • Step 3 - Type username and password
  • Step 4 - Click Login button
  • Step 5 - View Order History

At some point, the website changes, and makes the username and password entry, a separate action (this is quite common where a site supports SSO for example). It may be necessary to change the monitor as follows:

• My Monitor
  • Step 1 - Homepage
  • Step 2 - Click My Account
  • Step 3 - Type username
  • Step 4 - Type password
  • Step 5 - Click Login button
  • Step 6 - View Order History

In the above example, the fundamentals of the monitor have not changed (i.e. it’s the same monitor, and the user wants to maintain the history of the monitor, when looking at results).

This is true for:

• The monitor as a whole
  • i.e. I can see the history of results for this single monitor, before and after the change
• Each of the steps
  • i.e. I can see a history of each step, before and after the change
  • e.g. I can see the history of the View Order History step over time (on a chart), irrespective of it being step 5, or step 6

History

A performance chart might look something like this, showing performance over time, for where a step existed (and also for the total time of the whole monitor):

In this example, we’ve wanted to control what step is the same, before and after the change. We’ve decided to make the Type username and password different to the Type username and Type password steps. You can see that the performance results for the Type username and password series stops, and then they start for the Type username and Type password series.

However, there is continuity in the other series, even View Order History which is a step action after the username/password entry.

In other scenarios, the user may want to keep a step the same, even though the name has changed (i.e. it may not be appropriate to use the name as the unique factor in describing a step, as the user would lose history for that monitor, if they wanted to relabel the name of the step).

Sequence

The other requirement is that the result data should always store the relative order of the steps (their sequence). This is to ensure we can build charts that represent the performance of a monitor, e.g. a stacked column chart, or an area chart, where each step is represented by a different series. In this scenario, we need to know the order of the steps.

Potential Options

We need to balance giving users full control over this (e.g. imagine defining an id for every monitor and every step) as well as convenience (e.g. just assuming name is what maintains the step history, which means a name change will lose history). There may also be options for combining these in some way that gives more flexibility (e.g. name is mandatory, so assume this for the id too - if the name needs to change, then the user can force the id to the old name if they want to maintain history).

Requirements

The behaviour of making a change to the monitor must be deterministic, and defined in such a way that the user can make a change to steps and/or the monitor, whilst still having control over when it should maintain history or not, in as simple a way as possible. Sequence of step results is always needed in the results.

ACs

• When browser test results are stored, it must be possible to determine the sequence that the steps were executed in (regardless of the name, or if they change over time)
• It must be possible to configure a monitor in such a way that monitor history (monitor level results over time) is maintained, even if the behaviour or name of that monitor has changed (this is the user’s choice)
  • Similarly, it must be possible for the user to configure that the monitor is different, and should not keep history with the previous configuration
• It must be possible to configure a monitor in such a way that step history (step level results over time) is maintained, even if the behaviour or name of that step has changed (this is the user’s choice)
  • Similarly, it must be possible for the user to configure that the step is different, and should not keep history with the previous configuration

[paulb-elastic]

For reference, right now (using Monitor Management, and running a test through the service), changing the Type username and password step to become two separate Type username and Type password steps, looks like this in Lens (having made no changes in Lens):

Similarly, in Exploratory View:

In both cases, the original step stops and the two new steps don't appear.

In th following example, Exploratory View has been updated by adding in series for the two new Type username and Type password steps:

[paulb-elastic]

In the case of changing the name of the monitor (in Monitor Management), which was done between 18:20 and 18:25, it maintains history in the Uptime results view:

However it does not in Lens (where the monitor appears to have stopped):

and also does not in Exploratory View (where the monitor appears to have stopped):

Not that unsurprising in the Lens / Exploratory View examples, as you specifically select the monitor (and step) by name.

[paulb-elastic]

We are going to leave this based on the name (for monitor and step) for the time being