DOM1 specific websockets issues that began on Friday 2nd Nov

RobinL commented 5 years ago

ATOS had a networking outage early (7am-9am ish) on Friday 2nd November. Connectivity to Chrome and IE were restored around 10am, but Firefox could not connect to internet after COP on Friday.

Following restoration of service DOM1 users experienced issues with websockets manifesting in various ways:

Shiny Apps greying out immediate
Cannot use terminal in Jupyter
Cannot execute code in Jupyter

These are exactly the same problem we were experiencing when websockets were not working correctly - see https://github.com/ministryofjustice/analytics-platform/issues/40. However, this time the problems are only happening on DOM1 - everything's working fine on macbooks.

RobinL commented 5 years ago

Note that we know significant changes were made to the network - for instance on Friday we noticed that DOM1 whitelisted apps no longer work (i.e. DOM1 IP address ranges have changed)

RobinL commented 5 years ago

The web browser is failing to switch protocol from http to websockets both over TLS and over plain text which is apparent because we're not seeing the 101 switching protocols message (status code).

RobinL commented 5 years ago

capture 1 capture

RobinL commented 5 years ago

Ticket has been raised with ATOS ref number 931128 at 17:30 Mon 5th Nov.

davidread commented 5 years ago

FYI I think the IP whitelisting issue is separate - see https://github.com/ministryofjustice/analytics-platform/issues/79

RobinL commented 5 years ago

These issues seem to be resolved now. Closing.

Users should see 25/25 connected at our new websocket monitoring app: https://websocket-status.services.alpha.mojanalytics.xyz/

ministryofjustice / analytics-platform

DOM1 specific websockets issues that began on Friday 2nd Nov #78