New WMTS Notebook

[AaronV77]

This PR includes the notebook that was discussed in this mornings meeting. Thanks

[ceball]

Presumably the new notebook failed to run because we need to update dependencies.txt from quest 2.5.0 to a newer version? Is a new version likely to be packaged soon?

(Should we be putting quest onto conda-forge?)

[AaronV77]

@ceball I never got around to packaging Quest to a new version on Friday. I will ask @sdc50 about a new version tag and get a new package put on anaconda sometime before lunch. We are currently in the workings of trying to get Quest onto conda-forge, but we have some dependencies that are not on conda-forge and only support python 2.7. Once we can get those fixed we will be able to get Quest on conda-forge. Thanks

[AaronV77]

@ceball I just got Quest packaged for 2.6.0, and updated the dependency file to pull the newest version down. I think I replaced the version number in the correct place but if not, please let me know. Thanks

[ceball]

Thanks for doing that!

From what I remember of the demo of this notebook, my guess is that the tests are now failing just because the quantities being set by ipywidgets don't have (suitable) defaults, so when the notebook is run non-interactively, things don't work. Or something like that - I haven't opened the notebook yet to check that. I'd be happy to investigate and fix that up tomorrow (my workday's over for today).

[sdc50]

@AaronV77 I think it should be an easy fix to add a default bbox if one isn't specified in the widget, as is the case when the notebook is run in Travis.

There may be a better way (in which case I'd love to learn it), but I would suggest changing the following:

xs, ys = box_stream.element.array().T
bbox = list(gv.util.project_extents((xs[0], ys[0], xs[2], ys[1]), ccrs.GOOGLE_MERCATOR, ccrs.PlateCarree()))

to this:

if box_stream.element:
    xs, ys = box_stream.element.array().T
    bbox = list(gv.util.project_extents((xs[0], ys[0], xs[2], ys[1]), ccrs.GOOGLE_MERCATOR, 
    ccrs.PlateCarree()))
else:
    bbox = [-90.88, -32.35, -90.87, -32.36]

[AaronV77]

@sdc50 awesome suggestion, I never imaged that the notebook would be ran non-interactively. I will do this and add defaults to the ipywidgets like you mentioned yesterday as well. Thanks

[ceball]

@AaronV77, thanks. I assume you're finished with this PR now?

@philippjfr: are you ok with this being merged? (I'm fine with it; I can leave a note for Jim to review later and possibly suggest improvements, but it seems to me like we shouldn't hold up merging).

Chris to do:

• [ ] copy new quest package (and any updated dependencies) to pyviz/label/earthsim (we do this for a package like quest just to keep the channel list in the earthsim installation instructions as short as we can, so that the installation command doesn't get any longer than it already is...)
• [ ] clear notebook metadata

[philippjfr]

are you ok with this being merged?

Yes, I think I'll follow up with a PR that displays the downloaded tiff.

[AaronV77]

@ceball Yes I'm done with this, and the metadata should be cleared in this. Thanks

[jlstevens]

Glad to see this merged! I'll now work on adapting this notebook into the dashboarding work (PR to appear very soon).

[jlstevens]

@AaronV77 I might be confused and missing a transformation step or maybe there is a bug in the geotiff output tile.

If you set the following two lines before the options dictionary is defined (used by the call to quest.api.stage_for_download) and run the notebook:

bbox = [-73.6738, 45.9000, -67.9767, 48.7988]
zoom_widget_value.value = 7

Then a tile is fetched that contains Quebec City roughly in the middle. The issue is that the coordinates specified by the geotiff don't seem to include Quebec City. If at the end of the notebook I run:

# The tile has Quebec in the middle (can confirm by viewing geotiff directly)
import xarray as xr
da = xr.open_rasterio(list(meta.values())[0]['file_path'])
print('xmin {min} and xmax {max}'.format(min=float(da.x.min()), max=float(da.x.max())))
print('ymin {min} and ymax {max}'.format(min=float(da.y.min()), max=float(da.y.max())))
print('The lat/lon for Quebec is 46.947703, -71.210075 which is outside these ranges')

Which gives me this output:

xmin -56.246337890625014 and xmax -50.62866210937501
ymin 55.780931649674855 and ymax 60.23545253899177
The lat/lon for Quebec is 46.947703, -71.210075 which is outside these ranges

If I have access to the correct lat/lon coordinates for the tile, then I could slice the tile down to the requested bounding box. As discussed in out last meeting, this might also be useful functionality that quest could offer directly.

Either way, do you have any suggestions on how I could slice the tile to the specified bounding box?

[sdc50]

@jlstevens There must be a bug in the georeferencing process. I'll take a look into what is going on, and I think we can also easily add a clipping operation (using rasterio). We are hoping to have this pushed out with a new release of Quest this week.

[AaronV77]

@jlstevens after looking at your example I think I found the problem and its because your bounding box is x-min, y-min, and x-max, y-max. Quest needs it to be x-min, y-max, x-max, y-min. Thanks.

[jlstevens]

@AaronV77 I have the same problem using the bbox of bbox = [-73.6738, 48.7988, -67.9767, 45.9000] which I believe is using the order you are suggesting.

[AaronV77]

@jlstevens nope you are absolutely correct, there is something wrong with the calculation being done. For some reason that this didn't get posted to the form... @sdc50 has taken the plugin and is making the corrections. Like he said the updated basemap plugin will be in 2.6.1, and that will hopefully be out before the end of the week. One of us will post here when the new release is out. Thanks

[sdc50]

@jlstevens we released Quest 2.6.1 today. I can install it with the following command:

conda install -c conda-forge -c erdc quest

Let me know if you have any issues. Also, see this updated notebook for some changes in the interface: https://github.com/erdc/quest/blob/master/examples/notebooks/WMTS_Example.ipynb

[jlstevens]

@sdc50 Thanks for the heads up! I'll try it out now with the updated notebook.

[philippjfr]

Thanks @sdc50, works for me although it seems that because quest pins pandas<=0.22 I had to explicitly pin quest with:

conda install -c conda-forge -c erdc quest=2.6.1

[jlstevens]

It all seems to be working correctly now, thanks!

[AaronV77]

@philippjfr one of our dependencies has an issue with a deprecated Pandas function call. I have put a PR into the dependency for the fix. Once that PR has been accepted then we will be able to unpin the Pandas dependency. Thanks