`select_on_a_map` function occasionally does not respond to input

[robbibt]

This is going to be a tricky issue to bug-fix as it is unpredictable and only occurs sometimes:

Occasionally when the select_on_a_map function is run, the function allows users to select a polygon/rectangle of interest, but then does not allow the user to press "done" to continue and output the Geopolygon object. The "done" button does not respond to user input, and nothing happens when it is clicked.

When this is occuring, I've noticed that the white lat/lon box down the bottom left also typically appears blank instead of being filled with coordinates (it can however still occur when the box is not blank):

I haven't tested this exhaustively, but it seems like this issue is more likely to occur when a Dask client has been set up in a previous cell.

[robbibt]

@Kirill888 Sorry I don't have more precise steps to reproduce - I'll try and narrow it down, but it does happen pretty regularly - perhaps one in every 10 times the following code is run:

import os
import dask
from datacube.utils.dask import start_local_dask
from datacube.utils.rio import configure_s3_access
from odc.ui import select_on_a_map

# Configure dashboard link to go over proxy
dask.config.set({"distributed.dashboard.link":
             os.environ.get('JUPYTERHUB_SERVICE_PREFIX', '/')+"proxy/{port}/status"})

# Start up a local cluster
client = start_local_dask(mem_safety_margin=3)

## Configure GDAL for s3 access
configure_s3_access(aws_unsigned=True,  
                    client=client)

# Plot interactive map to select area
geopolygon = select_on_a_map(height='600px', 
                             center=(-26, 135), 
                             zoom=4)

[Kirill888]

@robbibt mem_safety_margin=3 will only leave 3 bytes of "safety margin". Did you mean '3G'?

This problem can happen when notebook "dies", or you whole sandbox is killed. Next time this happens can you please:

1. Check that some other notebooks can run (sandbox didin't die)
2. Interrupt Kernel (square "stop" button) and paste exception log here

[robbibt]

Whoops, yep, most of my testing has been using '3Gb', I entered it incorrectly on that small example only.

It's working well again now, but I'll do 1 & 2 as soon as it stops again.

[robbibt]

This is happening again:

1) Can confirm that other notebooks can run 2) Exception produced when interrupting kernel:

distributed.nanny - WARNING - Restarting worker
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-2-02dd76b09233> in <module>
      4 geopolygon = select_on_a_map(height='600px', 
      5                              center=(-26, 135),
----> 6                              zoom=4)

/usr/local/lib/python3.6/dist-packages/odc/ui/_map.py in select_on_a_map(**kwargs)
    234         return Geometry(state.selection, epsg4326)
    235 
--> 236     return ui_poll(lambda: extract_geometry(state) if state.done else None)

/usr/local/lib/python3.6/dist-packages/odc/ui/_ui.py in ui_poll(f, sleep, n)
     55     from jupyter_ui_poll import run_ui_poll_loop
     56 
---> 57     return run_ui_poll_loop(f, sleep, n=n)

/usr/local/lib/python3.6/dist-packages/jupyter_ui_poll/_poll.py in run_ui_poll_loop(f, sleep, n)
    111             yield x
    112 
--> 113     for x in with_ui_events(as_iterator(f, sleep), n):
    114         if x is not None:
    115             return x

/usr/local/lib/python3.6/dist-packages/jupyter_ui_poll/_poll.py in with_ui_events(its, n)
     79     with ui_events() as poll:
     80         try:
---> 81             for x in its:
     82                 poll(n)
     83                 yield x

/usr/local/lib/python3.6/dist-packages/jupyter_ui_poll/_poll.py in as_iterator(f, sleep)
    106         while x is None:
    107             if sleep is not None:
--> 108                 time.sleep(sleep)
    109 
    110             x = f()

KeyboardInterrupt: 

[Kirill888]

cool, thanks, it does look to be in the right place, which suggests that there were no exceptions in the callback.

this though: distributed.nanny - WARNING - Restarting worker, were you running anything on dask at the same time?

EDIT: actually callback code could very well crash and we won't see a thing, hm

[Kirill888]

What about map itself, can you draw another rectangle or does it stop responding when that happens?

[robbibt]

Yep, I can draw/delete/clear multiple other rectangles, the "done" button just doesn't respond:

this though: distributed.nanny - WARNING - Restarting worker, were you running anything on dask at the same time?

In this example, I simply set up a local dask client in the first cell of the notebook (so that I can use dask to load data after the interactive map selection). The interactive map selection is the first thing that happens after starting the client, and the task stream doesn't show any processes running.

[robbibt]

Here's a video of the issue occuring:

https://www.loom.com/share/d85a0c3411c14e36984c7320bc256355

[Kirill888]

Can you share exact (hopefully minimal) notebook and how often does it happen? Also which sandbox environment are you using? I need to be able to reproduce it to debug it.

[robbibt]

Minimal example which is currently working correctly for me but wasn't working about 10 minutes ago: select_on_a_map.zip

I'm using the "Development environment - Jupyterhub environment with 8 Cores, 60G Memory" on the DEA Sandbox. I've now uninstalled all user libs except:

coverage 4.5.4  
nbval    0.9.4