SECOORA assets map

[ocefpaf]

@vembus mentioned that we should produce an updated map displaying all the SECOORA buoys, gliders, radars, etc. We can probably pull the positions and some other metadata from a catalog search and re-run it weekly to get an updated version of the map.

[vembus]

Kyle has set up one for us for the proposal, I wanted to check as to best way to have this set up, as often we need such a map requested for proposal etc.

[vembus]

Need to get this revised updated etc.

[ocefpaf]

Thanks! That means I have a place to start.

Ping @kwilcox. Can you share your experiences/current way to produce the figure @vembus mentioned?

[ocefpaf]

@vembus

Please take a look at this notebook below:

http://nbviewer.ipython.org/github/ioos/secoora/blob/master/AbstractsPostersPresentations/SECOORA_assesets_map/SECOORA_assets_map.ipynb

If that is OK we now have a way to go from the Excel file to GeoJSON objects.

The GeoJSONs can use to create the assets map (interactive or static).

[vembus]

Thanks, Looks neat. Vembu

Sent from my Verizon Wireless 4G LTE smartphone

-------- Original message -------- From: Filipe notifications@github.com Date: 10/14/2015 12:52 PM (GMT-05:00) To: ioos/secoora secoora@noreply.github.com Cc: Vembu Subramanian vembu@secoora.org Subject: Re: [secoora] SECOORA assets map (#215)

@vembus

Please take a look at this notebook below:

http://nbviewer.ipython.org/github/ioos/secoora/blob/master/AbstractsPostersPresentations/SECOORA_assesets_map/SECOORA_assets_map.ipynb

If that is OK we now have a way to go from the Excel file to GeoJSON objects.

The GeoJSONs can use to create the assets map (interactive or static).

— Reply to this email directly or view it on GitHub.

[rsignell-usgs]

@ocefpaf, just for fun, I tried dropping one of the json output files from your notebook into the TerriaJS-based portal (http://tampabaymap.org) that @metanomy set up, and it worked just fine:

[ocefpaf]

@rsignell-usgs

Cool! The idea is to find an easy way to go from an updated Excel spreadsheet to an assets map. I guess we are almost there. I still need to save all polygons as one GeoSJON storing the station name in the file. Then you could plot everything in one step.

[metanomy]

@ocefpaf If there is a way to invoke the execution of your notebook through an API, I can test adding that data output as a layer in our portal, with the idea being that anytime a user selects the layer, the notebook executes and returns dynamic data.

[vembus]

Cool!!

Sent from my Verizon Wireless 4G LTE smartphone

-------- Original message -------- From: Rich Signell notifications@github.com Date: 10/14/2015 1:15 PM (GMT-05:00) To: ioos/secoora secoora@noreply.github.com Cc: Vembu Subramanian vembu@secoora.org Subject: Re: [secoora] SECOORA assets map (#215)

@ocefpaf, just for fun, I tried dropping one of the json output files from your notebook into the TerriaJS-based portal (tampabaymap.org) that @metanomy set up, and it worked just fine:

— Reply to this email directly or view it on GitHub.

[rsignell-usgs]

@metanomy , we once had a plan (@vembus, @ocefpaf and I) to turn some of these workflows into custom services using flask, basically passing the parameters needed to run the script as parameters in the URL request to the service. Our first service was going to be the water level comparison service, where the user specifies a bounding box and a time extent, and gets back the water level time series from models and data as JSON. But we never got around to it.

Do you know how to do this stuff? Basically take an existing python code and turn it into a service?

[kwilcox]

@ocefpaf did you see this? https://github.com/SECOORA/static_assets

[metanomy]

@rsignell-usgs This is where we want Jupyter. It gets even better - I think we can integrate this library (http://jupyter.org/jupyter-js-services/) into TerriaJS so that notebooks become just another catalog type.

[ocefpaf]

@ocefpaf did you see this? https://github.com/SECOORA/static_assets

@kwilcox Nope. I pinged you before starting this to avoid re-inventing the wheel! Too late :unamused:

@ocefpaf If there is a way to invoke the execution of your notebook through an API

@metanomy I do not have experience building these kind of services, but it is possible and I do want to start doing things that way. (And I guess it will relatively easy based on other people experiences.)

[ocefpaf]

@rsignell-usgs and @metanomy I guess that TerriaJS can take both shapefiles and GeoJSONs, correct?

My question is: what is the preferred format to save these features?

PS: I usually prefer GeoJSONs for its simplicity and the fact that is one file instead of 4 (.shp, .shx, .dbf, and .sbn)

[metanomy]

@ocefpaf GeoJSON is great for vector data. TerriaJS can even do CSV (with "lat" and "lon" as the headers for latitude and longitude values). If you have time-series data, TerriaJS will animate (with a time slider, no less) the CSV data, as long as the column with the time stampe has the word "time" in the header.

[ocefpaf]

@kwilcox I see you are using different values for the radar range.

mhz_to_dist = {
    5:    225,
    8.3:  175,
    12.6: 125,
    16:   100,
}

Mine are from this GMT script @vembus sent me.

@vembus that information should be added to the spreadsheet to avoid a world wide economic recession :wink:

[ocefpaf]

@ocefpaf GeoJSON is great for vector data. TerriaJS can even do CSV (with "lat" and "lon" as the headers for latitude and longitude values). If you have time-series data, TerriaJS will animate (with a time slider, no less) the CSV data, as long as the column with the time stampe has the word "time" in the header.

@metanomy thanks! I looked into TerriaJS some time ago and all I remember is that I found it awesome, but never got time to actually build anything using it. I guess it is time to fix that!!

[metanomy]

@ocefpaf @rsignell-usgs I think we can use Falcon (http://falconframework.org/) to build an API wrapper around an iPython notebook. I think the only part that I'm unsure of is how to configure the notebook to save the Geojson as a file that the API can then return to the calling client.

[ocefpaf]

@ocefpaf @rsignell-usgs I think we can use Falcon (http://falconframework.org/) to build an API wrapper around an iPython notebook.

Thanks @metanomy falcon seems an interesting framework. However, I think we can make things simpler and even drop the notebook. (I only use notebook to show literate versions of my scripts.)

@kwilcox approach, for example, is virtually 100% standard library and probably might lighter and more adequate than mine. Mine requires pandas, xlrd, matplotlib, and shapely!

The only advantage of my approach is to avoid the Excel to CSV conversion. That would allows us to host the spreadsheet online (Google Docs for example. That way people can easily update the information there) and keep running the script to yield updated JSON features.

[metanomy]

@ocefpaf I'm happy to help test your script with tampabaymap.org, if that would be helpful. I still think there is value with putting an API around a notebook, especially for some of the more complicated uses cases that @rsignell-usgs has explored (multi-catalog query, mult-model comparison, etc).

[ocefpaf]

I still think there is value with putting an API around a notebook, especially for some of the more complicated uses cases that @rsignell-usgs has explored (multi-catalog query, mult-model comparison, etc).

Oh yes! I am VERY interested in that as well. (In fact I that high priority in my TODO.)

I meant just the current case deserves a simpler solution.

[metanomy]

@ocefpaf Another option is to install this script to the Google Sheet (https://github.com/mapbox/geo-googledocs/), and then anytime someone updates the data, they just regen the GeoJSON.

[kwilcox]

There is already an OGC service standard for vector data (WFS). I'm assuming TerriaJS can support requesting GeoJSON from a GeoServer based WFS? We should work on setting up the workflow for getting the Google Spreadsheet into SECOORA's GeoServer and push people to that. It will also show up in the SECOORA Data Portal (searchable and discoverable by anyone).

[rsignell-usgs]

Yes, TerriaJS can consume data from an WFS endpoint (that's one of the 1st things I tried).

To test this for yourself, try going to http://nationalmap.gov.au, click the "+Add data" button and drop this WFS endpoint URL: https://www.sciencebase.gov/catalogMaps/mapping/ows/5342c5fce4b0aa151574a8ed You should see this appear under "User-Added Data" at the bottom of the dataset list. Select a layer, then click on "Now Viewing" and click on "Zoom To". You should see this:

[metanomy]

@ocefpaf Is there any reason the data in the spreadsheet shouldn't be hosted as a layer inside a geospatial server (like Geoserver) as @kwilcox suggests? You could even use GeoExplorer (or any other WFS-T client) to edit the data as necessary.

[ocefpaf]

No reason. It definitely should!

[ocefpaf]

@vembus

This is a proof of concept:

http://nbviewer.ipython.org/github/ioos/secoora/blob/master/AbstractsPostersPresentations/assets_map/assets_map.ipynb

It is not as pretty as @kwilcox's version and there are some things yet to be done. However, this version is hackable, and everything used to construct the map is in the repository. So it is relatively easy to modify and update it.

Things to note:

• I am using text popups to show the assets info. A static version needs the text plotted as a map overlay. That can be tricky though. It is very easy to make a mess and pollute the map.
• I had to re-create some of the icons. Both @kwilcox's repository and the resources you sent me via e-mail did not had all icons in all colors. Hopefully the differences are small not noticeable.
• I do not have any info on the big gray triangles from the original map. @kwilcox called them georef_glider_triangles. Are they glider deployment and recovery positions?
• I do not have any other glider info to plot.
• I did not create the legend yet. If that figure will be used in a text I'd rather not have it and write the legend instead. If the figure will be used online or as a slide we need to add the legend.

To save a static version we have two choices:

1. Open the HTML in full screen, zoom in/out, navigate to the place where we want, and save the image.
2. Use cartopy to plot a PNG version.

Option (1) requires some thinking about the text placement. And (2) a lot of tweaking to get things in place.

PS: We have a few options for the background map. (See here for some options.)

[ocefpaf]

Here is a PNG version with a stolen legend:

[ocefpaf]

@vembus the link below has a full screen interactive version of the map.

http://ocefpaf.github.io/secoora_assets_map/secoora_assets.html

@kwilcox I went ahead and create a separate repo here. I will add the updates there.

[metanomy]

@vembus Are you running any kind of WMS/WFS server that could host this data? It would be (selfishly) helpful to display on tampabaymap.org. But also nice for others to ingest for their data, too.

[kwilcox]

@metanomy SECOORA runs a Geoserver that can be used

[metanomy]

@kwilcox Awesome - is that publically accessible?

[kwilcox]

I haven't been following this closely enough! The original map was produced using Mapbox, solely because of the labeling requirements. The labels were all somewhat manually placed where you see them; Mapbox supports that while most other services don't or the support was too limiting.

Mapbox was a short term solution. The long term is to setup a periodic Drive to GeoServer converter (GIS format in between). Using SLDs we should be able to get relatively close to the original. Once in Geoserver it will be an public WMS/WFS layer.

[metanomy]

Let me know if there is anything Metanomy can do to help. As soon as the check clears, we'll be an official member of SECOORA.

[ocefpaf]

@kwilcox I want your opinion on something.

I wanted to go from the Excel spreadsheets to GeoJSONs features:

• MultiLinestring for glider
• MultiPoint to a each set of observations
• MultiPolygon+ for the hf-radar ranges wedge.

Leaflet can add a PNG icon to each Point afterwards and we can easily generate and update the map.

Do you think this is worth pursuing? Or do you have other plans?

[metanomy]

@ocefpaf I found a great site that made it easy to convert GeoJSON to SHP (helpful for import into Geoserver, although you should be able to do a direct GeoJSON import through the REST API): http://www.mapshaper.org/

[ocefpaf]

@metanomy we can do a shapefile directly from python. I just like GeoJSON better for its readability.

I just don't want to go ahead and do something again just to find out that @kwilcox already did it.

[ocefpaf]

@kwilcox I cleaned up my scripts and updated the data. Maybe that is useful for you...

I am converting the spreadsheets to GeoJSONs and Shapefiles with data_frame2gis.py.

The GeoJSONs for stations and hfradars have some defaults for color, fill, alpha, icon, popup, etc in "properties." I plan to read that from the folium side to easy the map creation. (See https://github.com/ocefpaf/secoora_assets_map/issues/1.)

[ocefpaf]

I plan to read that from the folium side to easy the map creation.

Almost there. Next version of folium has custom icons. Parsing the GeoJSON to create the assets map became much easier (and less hackish) now:

http://nbviewer.ipython.org/github/ocefpaf/secoora_assets_map/blob/gh-pages/test_geojson.ipynb

NB: We still need to read the simplestyle-spec to get the right automatic GeoJSON styling.

[ocefpaf]

@kwilcox merged our efforts in https://github.com/SECOORA/static_assets