[Analytics] Raises tracebacks when network connection is lost

[jacobtomlinson]

Describe the bug If I start a cluster and enable Coiled Analytics and then lose internet connection I get a traceback printed out every time the interval happens.

from dask.distributed import Client
client = Client()
import coiled.analytics
coiled.analytics.register()

# Turn off WiFi

Failed to send message to route: metrics
Traceback (most recent call last):
  File "https://cloud.coiled.io/api/v2/analytics/preload", line 176, in post_request
tornado.simple_httpclient.HTTPTimeoutError: Timeout while connecting
2022-07-13 14:32:34,171 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop obje
ct at 0x7f9ca98d4f10>>, <Task finished name='Task-2130' coro=<CoiledTelemetry.send_metrics() done, defined at https://cloud.coiled.io/api/v2/analytics/preload:289> exception=HTTP 599: T
imeout while connecting>)
Traceback (most recent call last):
  File "/Users/jtomlinson/miniconda3/envs/dask-analytics/lib/python3.10/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/Users/jtomlinson/miniconda3/envs/dask-analytics/lib/python3.10/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "https://cloud.coiled.io/api/v2/analytics/preload", line 358, in send_metrics
  File "https://cloud.coiled.io/api/v2/analytics/preload", line 176, in post_request
tornado.simple_httpclient.HTTPTimeoutError: Timeout while connecting
Failed to send message to route: metrics
Traceback (most recent call last):
  File "https://cloud.coiled.io/api/v2/analytics/preload", line 176, in post_request
tornado.simple_httpclient.HTTPTimeoutError: Timeout while connecting

Issue Priority, optional

• [x] Low
• [ ] Medium
• [ ] High

[ncclementi]

@ian-r-rose would you mind taking a look at this one?

[ian-r-rose]

Thanks for the report @jacobtomlinson! I'll take a look.

[dchudz]

I wonder what the desired behavior is.

Maybe a shorter, less frequent note? Or just a single message that analytics is off b/c we lost the connection to Coiled?

[jacobtomlinson]

As a user, I think I would never want to hear from coiled analytics ever. Maybe there could be some way of checking if it's ok in case you look at the dashboard and don't see anything and want to check. But that should be a method you can call to explicitly ask.

But in 99% of cases, I want folks in my group to turn on analytics and then never have to think about it again. I definitely don't want them to see random errors that are nothing to do with what they are doing. The data is useful to me, not them.

[jacobtomlinson]

Or just a single message that analytics is off b/c we lost the connection to Coiled?

This would probably be ok, but even then I kinda don't want to see this.

[shughes-uk]

Analytics plugin should be fairly quiet now it was rewritten.