search

elastic / uptime

This project includes resources and general issue tracking for the Elastic Uptime solution
12 stars 3 forks source link

Perform upgrade test from 7.16 to 8.0.0 #363

Closed andrewvc closed 2 years ago

andrewvc commented 3 years ago

Aside from https://github.com/elastic/kibana/issues/112757 there are no known issues WRT upgrading from 7.16 to 8.0.0. We should manually test this flow however and see if we're missing anything. Ideally we'd setup demo monitors for each protocol in 7.x then do the upgrade and poke around.

dominiqueclarke commented 2 years ago

I've begun testing this.

Test Results

There were no glaring errors while performing this test plan. App is working as expected.

Setup

  1. Create a 7.16.1 cluster using cloud.elastic.co in the Los Angeles (us-west2) region.
  2. Create demo monitors using heartbeat yml syntax for each protocol.
  3. Create demo monitors in fleet for browser inline, browser zip, TCP, and HTTP using the elastic-agent-complete:7.16.1 image.
  4. Upgrade to 8.0.0-beta1 via the cloud console

Test Plan

Uptime overview

  1. Navigate to Uptime
  2. Paginate all the way through results
  3. Use all available filters: location, port, schema, and tag
  4. Use search bar, KQL and basic
  5. Change time on DatePicker
  6. Filter by tags by clicking on the tag itself
  7. Scrub the overall ping histogram chart
  8. Scrub the individual ping histogram chart in the monitor row
  9. Create a simple alert using the alert toggle
  10. Expand the accordion of the monitors and check for details (last error, connected alert rules)

Monitor details

  1. Visit the details page of each schema, HTTP, ICMP, TCP, and browser
  2. Use all available filters on the monitor duration page
  3. Scrub the monitor duration and ping histogram charts on each details page
  4. Create ML job from monitor duration chart

Alerts

  1. Create ML alert from monitor duration chart
  2. Create TLS and Monitor Status Alerts, with filters

Browser monitor check details page

  1. Visit check details page for browser monitors
  2. Visit the performance breakdown page (waterfall chart), use filters and click bars
  3. Navigate between checks on the performance breakdown page
  4. Navigate between steps on the performance breakdown page

Settings

  1. Navigate to Settings Page
  2. Save TLS or Heartbeat index settings
  3. Create and save a default connection

Exploratory View

  1. Navigate and configure Exploratory view

@paulb-elastic @andrewvc Please note, I this test was performed for the upgrade between 7.16.1 and 8.0.0-beta1 as 8.0.0-beta1 is stable in cloud. Do you think this test needs to be performed for 8.0.0-SNAPSHOT as well? If so, I can perform an upgrade locally.

One thing that is different between 8.0.0-beta1 and 8.0.0-SNAPSHOT that may be significant is the change to our default index settings, which updates the defaults to heartbeat-7*, heartbeat-8*,synthetics-*. While those defaults were not in place for this test, I did manually update my settings and it worked just fine with above settings.

Also please note, I did not test User Experience. Is that in scope for this ticket?

paulb-elastic commented 2 years ago

Thanks for going through this and providing the details @dominiqueclarke. Out of interest, have we also run the automation (or is that automatically done as part of the 8.0.0-beta1 build)?

I also think it’s worth checking User Experience, just to be sure (thanks for reminding us of that).

Regarding 8.0.0-beta1 vs 8.0.0-SNAPSHOT. Are you specifically referring to https://github.com/elastic/kibana/issues/112757, which seems to have been merged on 18/Nov, after beta.1? The fact you manually tested this is probably fine. It’s just if we think it’s worth retesting that part when 8.0.0-rc1 comes out?

dominiqueclarke commented 2 years ago

@paulb-elastic automation runs hourly so automation would have been run before 8.0.0-beta1 was built.

dominiqueclarke commented 2 years ago

User Experience

A test upgrade was performed in cloud on the cloud first testing region from 7.16.2 to 8.0.0-beta1 for User Experience. After the upgrade, the User Experience app ceased to show data. After investigating, APM Server was dropping unsampled transactions in 8.0.0-beta1 causing a broken experience in the User Experience app.

Sampled transactions should no longer be dropped in 8.0.0-SNAPSHOT. A test was attempted for 8.0.0-rc1, but the Kibana build appeared unstable as Fleet reported 'APM not found' error when navigating to the APM Integration.

Finally, a test was attempted in cloud for 8.0.0-SNAPSHOT and apm-rum-js-agent version 5.6.3. Unsampled transactions were not being dropped and User Experience was working as expected, including:

https://github.com/elastic/kibana/issues/122404 was identified, but that impacts 7.16.2 as well as 8.0.0.

dominiqueclarke commented 2 years ago

One thing that is different between 8.0.0-beta1 and 8.0.0-SNAPSHOT that may be significant is the change to our default index settings, which updates the defaults to heartbeat-7, heartbeat-8,synthetics-*. While those defaults were not in place for this test, I did manually update my settings and it worked just fine with above settings.

@paulb-elastic Here I was referring to elastic/kibana#112757. This has been tested in 8.0.0-rc1 and is working as expected with heartbeat version 7.16.2.

paulb-elastic commented 2 years ago

Thank you @dominiqueclarke for running through all the testing, especially with the various other issues that made this more complex.