Explorer UI seems to be operational even if local cluster has been destroyed in meantime

[thehejik]

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible):

• deploy a single-node local RKE cluster with rancher/rancher:master-head 51d074b
• In browser login to rancher Explorer UI and let the tab opened
• destroy/shutdown the cluster infrastucture - eg. terminate/destroy AWS EC2 instance running the single-node local cluster

Result:

• Even if the local cluster has been destroyed user can still browse Explorer UI (in tab opened before) without a sign that the cluster is actually down.
• Entries like GLOBAL APPS/Cluster Management and EXPLORE CLUSTER/local are still accessible when opened from /dashboard/home page
• local cluster still reported as Active on /dashboard/home overview page

Expected Result:

• User should be informed somehow that the local cluster is down and browsing thru web ui shouldn't be possible

Other details that may be helpful:

• In Ember UI user gets ERR_ADDRESS_UNREACHABLE in browser after a while when operating with UI

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): rancher/rancher:master-head 51d074b
• Installation option (single install/HA): single/HA

Cluster information

• Cluster type (Hosted/Infrastructure Provider/Custom/Imported): local (eg RKE1)
• Machine type (cloud/VM/metal) and specifications (CPU/memory): any
• Kubernetes version (use kubectl version): any

[richard-cox]

We get updates to known resources pushed to the client via web socket, there's no manual refresh client side. So if the local cluster goes down we won't know about it until either the sockets can't reconnect or requests for unseen resource types are made.

We could use the new and improved growl system to show a warning if the sockets can't reconnected after x attempts/y seconds, 'Failed to connect... some information may be stale, etc'?

Edit - It looks like Cluster Manager just signs the user out if there's issues. Maybe that's a second step?