search

HDFGroup / hdf-compass

Python-based viewer for HDF5 on other file formats
Other
130 stars 26 forks source link

Visualization of geographic arrays #118

Closed giumas closed 8 years ago

giumas commented 8 years ago

The BAG plugin has some geographic arrays to visualize. I was thinking to add something like a GeoArrayFrame that, for example, shows the axes with 'natural' coordinates and color-maps better suited for bathymetric grid. Is this a good idea?

jreadey commented 8 years ago

I don't have much experience with geographic plotting toolkits. It seems many of them are tricky to setup (and we need to think about the PyInstaller aspect).

How about just matplotlib? E.g. http://matplotlib.org/basemap/users/examples.html.

hyoklee commented 8 years ago

I like this idea.

kyang2014 commented 8 years ago

I agree. A simple approach may cover the majority.

From: H. Joe Lee [mailto:notifications@github.com] Sent: Tuesday, October 20, 2015 4:43 PM To: HDFGroup/hdf-compass Subject: Re: [hdf-compass] Visualization of geographic arrays (#118)

I like this idea.

— Reply to this email directly or view it on GitHubhttps://github.com/HDFGroup/hdf-compass/issues/118#issuecomment-149711870.

giumas commented 8 years ago

@jreadey Looking ahead I would prefer to use Cartopy (http://scitools.org.uk/cartopy/) for several reasons. Some of them are mentioned here: https://news.ycombinator.com/item?id=10009098

From a PyInstaller point of view, I have recently experienced that a 'basemap' application gains 100 MiB in size since it has to ship all the auxiliary maps that are installed with this package.

Said this, I would give a try to Cartopy for visualize the BAG bathymetry (so I am changing the label from 'question' to possible 'enhancement').

There is also another related question: do we want to introduce a mechanism similar to the plugin model store for the visualization frames? I ask this because, in my opinion, the plugin mechanism is currently not fully implemented.

For a new plugin:

About this last point, as use case, I would like to add some export functions to the BAG plugin (e.g., to convert some data sets in the BAG file to ESRI ascii grid), but this would require to sub-class the ArrayFrame.

@jreadey Am I correct or am I missing a different way to reach my aim?

jreadey commented 8 years ago

@giumas - cartopy sounds interesting, but I'm not keen to triple the size the size of the application for a feature that only some people will use. Can the maps be off-loaded somehow? Idea is that they would only be copied when they were accessed for the first time. I feel we're getting into something they may be best to hold off till the next release.

Re: the frames - I think it would be better for usability if new top-level windows were only created when the user selected Window|New Window or such. With the ArrayFrame/KeyValueFrame/Matplot lib windows all floating around it becomes hard for the user to keep track of everything. Can we adapt the code so that (for example) selecting a dataset displays the ArrayFrame in the original top-level window?

Would this approach improve the plugin-in model?

On your last point - the Compass approach should be to make it easy for plugin developers to write plugins. By easy I mean they don't need to touch code outside the plugin itself (or as little as possible - re your earlier comment), and that they don't need to know wxPython to write a plugin. This will limit the scope of what can be done using Compass, but developers always have the option of writing their own application.

Can you think of a clever way of adding plugin-specific menu items keeping the above in mind?

giumas commented 8 years ago

cartopy sounds interesting, but I'm not keen to triple the size the size of the application for a feature that only some people will use.

I believe that my point was not clear. Your suggested mpl_toolkits.basemap (not cartopy) has the 100-MiB issue:

jreadey commented 8 years ago

I misunderstood, I thought it was cartopy that was the 100mb.

Have you done a pyinstaller install with a cartopy app? What did the final size turn out to be?

giumas commented 8 years ago

I feel we're getting into something they may be best to hold off till the next release.

I agree on it. So I have just created a new generic milestone label 0.6.x.

jreadey commented 8 years ago

Ok. Getting the initial BAG plugin done for 0.6 is a good first step.

giumas commented 8 years ago

Re: the frames - I think it would be better for usability if new top-level windows were only created when the user selected Window|New Window or such. With the ArrayFrame/KeyValueFrame/Matplot lib windows all floating around it becomes hard for the user to keep track of everything. Can we adapt the code so that (for example) selecting a dataset displays the ArrayFrame in the original top-level window?

I personally like the current behavior.. However, a possible enhancement in this direction could be a menu item flag to switch between the 2 behaviors, similar to the Windows 'Browse folders' options:

capture

Would this approach improve the plugin-in model?

I am not sure whether this change could improve the model. Do you have an use case?

giumas commented 8 years ago

On your last point - the Compass approach should be to make it easy for plugin developers to write plugins. By easy I mean they don't need to touch code outside the plugin itself (or as little as possible - re your earlier comment), and that they don't need to know wxPython to write a plugin.

@jreadey : My idea would be more on only having available as option so that I could push a new Frame (e.g., GeoArrayFrame). This does not imply that you need to use it to write a plugin.

I believe that something of this kind has been around in the past: https://github.com/HDFGroup/hdf-compass/blob/develop/hdf_compass/compass_model/model.py#L408-L430. As far as I am aware this Plottable class is currently unused. Without a working example is unclear to me how it should work. Does @andrewcollette have some insights on such a class?

I am not a big fan of unused code being around without a working example. So it could be a candidate to be deleted.. as well as this commented part: https://github.com/HDFGroup/hdf-compass/blob/develop/hdf_compass/compass_viewer/frame.py#L166-L172

andrewcollette commented 8 years ago

@giumas, as far as I remember the Plottable class is an unused historical relic.

I can provide some development perspective on the plugins idea. The core concept I started with for HDF Compass is that the application provides a fixed set of top-level window classes, one for each flavor of Node. That strict decoupling means the GUI debugging only has to be done once, and people can implement their Node subclasses and other "plugin" things without fiddling with wxPython.

If additional types of frames will be added, I would suggest they be added to the core application following this approach; for example, a Map Node with a well-defined interface corresponding to a MapView frame.

One last thing: although these are called "plugins" I originally did not envision them as traditional plugins installed by end users. Rather, the original (commercial) view for HDF Compass was that all "plugins" and frames be distributed with the monolithic app. I think the HDF5 library misadventures with pluggable filters shows how valuable it is for such an app to take responsibility for aggregating and distributing such things. That's why, for example, there is currently no loading mechanism for plugins. But I realize the HDF Group may have different goals.

giumas commented 8 years ago

@andrewcollette Thank you! Your comment is very useful for my understanding of the full project.

@giumas, as far as I remember the Plottable class is an unused historical relic.

I will push a PR to delete the relic.

The core concept I started with for HDF Compass is that the application provides a fixed set of top-level window classes, one for each flavor of Node. That strict decoupling means the GUI debugging only has to be done once, and people can implement their Node subclasses and other "plugin" things without fiddling with wxPython.

I believe that this approach is very good. It has been a key of the success of the application.

If additional types of frames will be added, I would suggest they be added to the core application following this approach; for example, a Map Node with a well-defined interface corresponding to a MapView frame.

This is exactly what I need for the BAG file format (and I believe that would be beneficial for other models as, for instance, the ascii grid). I can obtain the same result by sub-classing externally to the HDF Compass, but I would like to keep such a type of 'hack' for special and limited cases.

One functionality that have been requested for the BAG plugin is to provide an 'export' option to save the array as geotiff or ascii grid (by calling hydroffice.bag methods that use GDAL). What is the general feeling on this? Does it classify as 'special case' that I need to manage externally to the HDF Compass?

One last thing: although these are called "plugins" I originally did not envision them as traditional plugins installed by end users. Rather, the original (commercial) view for HDF Compass was that all "plugins" and frames be distributed with the monolithic app. I think the HDF5 library misadventures with pluggable filters shows how valuable it is for such an app to take responsibility for aggregating and distributing such things. That's why, for example, there is currently no loading mechanism for plugins. But I realize the HDF Group may have different goals.

Given that now there is a namespace (hdf_compass), a simple but effective way to load the installed plugins would be to look in such a namespace for packages named '*_model' and then import them. However, I totally understand your point (and the risk of making this mechanism available). It might be worthwhile to underline the adopted solution since the term plugin can be misleading. Or, at least, it was for me since when I started to look at the project I was looking for a more traditional mechanism.

hyoklee commented 8 years ago

After playing with HDFCompass, I don't know why this application is still called "hdf-compass". It should be called "compass-viewer". HDF is just one of data formats that fit into Node-oriented compass model. It has a great potential as a general / flexible / extensible "data viewer" for all data formats that Python can support but the name starting with "hdf-" misleads and discourages other developers to take a serious look.

kyang2014 commented 8 years ago

One functionality that have been requested for the BAG plugin is to provide an 'export' option to save the array as geotiff or ascii grid (by calling hydroffice.bag methods that use GDAL). What is the general feeling on this? Does it classify as 'special case' that I need to manage externally to the HDF Compass?

This is a special case(export as geotiff or asci grid) for general HDF5 files but a more or less general case for HDF5 files in the Earth Science domain. It may be worth to have another layer to group plugins like BAG or ASCII Grid.

One last thing: although these are called "plugins" I originally did not envision them as traditional plugins installed by end users. Rather, the original (commercial) view for HDF Compass was that all "plugins" and frames be distributed with the monolithic app. I think the HDF5 library misadventures with pluggable filters shows how valuable it is for such an app to take responsibility for aggregating and distributing such things. That's why, for example, there is currently no loading mechanism for plugins. But I realize the HDF Group may have different goals.

I also like the idea distributing everything with one monolithic app. Having end users to install each plugin limits the usability.

— Reply to this email directly or view it on GitHubhttps://github.com/HDFGroup/hdf-compass/issues/118#issuecomment-150323248.

giumas commented 8 years ago

After playing with HDFCompass, I don't know why this application is still called "hdf-compass". It should be called "compass-viewer". HDF is just one of data formats that fit into Node-oriented compass model. It has a great potential as a general / flexible / extensible "data viewer" for all data formats that Python can support but the name starting with "hdf-" misleads and discourages other developers to take a serious look.

@kyang2014 This is a comment that was in my mind since a while, but I stopped myself from writing it since it could have appeared as 'outside of my scope'.

I have recently added HDF5 label with the intent to underline the tickets that are specific for the HDF5 plugin.

jreadey commented 8 years ago

I'm with @andrewcollette - having frozen binaries that include all the plugins add real value to users.

Re: the 'HDF' aspect of HDFCompass - though almost any file format could be supported by a plugin, I think the model favors plugins that are HDF5 domain formats or file formats that are "HDF-like" in the sense that they have array-based data format and optionally, some type of grouping mechanism.

Or if you don't buy that line of reasoning, think of it as HDFCompass as "the Compass viewing tool provided by the HDF Group (with contributions from others)".

giumas commented 8 years ago

This is a special case (export as geotiff or asci grid) for general HDF5 files but a more or less general case for HDF5 files in the Earth Science domain. It may be worth to have another layer to group plugins like BAG or ASCII Grid.

@jreadey what are your thoughts on this? An implicit implication of adding a Map node are that gdal, proj4 and (maybe) geos functionalities/libraries/data must also be added to the frozen application. However, I have already done it in the recent past.

andrewcollette commented 8 years ago

Regarding the data-export functionality, I didn't have time to implement it but when designing HDF Compass I did have an "export plugin" functionality in mind. This would be the inverse of the plugin classes that take on-disk files (for example) and produce Nodes. A hypothetical interface might look like this:

class Exporter(object):

    classkind = "XYZ File Format"
    file_extension = '.xyz'

    @staticmethod
    def canhandle(node):
        """ Determine if this exporter can transform the given Node into
        a particular file format (T/F) """
        pass

    def __init__(self, node):
        """ Create an instance that can export the data in "node" to disk. """
        pass

    def export(self, output_url):
        """ Export the data from the given node to disk """
        pass

These could be registered in a similar fashion to the various Stores. An "Export" menu would be populated with the registered exporters; those for which canhandle() is False for the currently displayed node greyed out.

A simple CSV exporter, for example, might accept any Array implementation with two or fewer axes.

jreadey commented 8 years ago

I think for a CSV exporter (ASCII Grid or GeoTIFF likewise I believe) it would be problematic to take an entire Compass model and some how shoe horn it into a CSV file. This seems more like a command you would perform on a specific array (something like right-click save as CSV). The same code could be used to copy to the clipboard (#21).

For a generic data-export as @andrewcollette is discussing, the target file format would need to be able to support the general Compass model. I'm not sure that leaves many options there are other than HDF4 and HDF5 (and I don't see much value in a HDF4 exporter). Would it be simpler to ditch the export-plugin concept and just have an HDF5 exporter?

kyang2014 commented 8 years ago

In practical sense, I don’t see it has much value(at least for now) to export as HDF5 although it is fit to the Compass model. CSV or GeoTIFF may have more users.

From: John Readey [mailto:notifications@github.com] Sent: Thursday, October 22, 2015 3:28 PM To: HDFGroup/hdf-compass Cc: Kent Yang Subject: Re: [hdf-compass] Visualization of geographic arrays (#118)

I think for a CSV exporter (ASCII Grid or GeoTIFF likewise I believe) it would be problematic to take an entire Compass model and some how shoe horn it into a CSV file. This seems more like a command you would perform on a specific array (something like right-click save as CSV). The same code could be used to copy to the clipboard (#21https://github.com/HDFGroup/hdf-compass/issues/21).

For a generic data-export as @andrewcollettehttps://github.com/andrewcollette is discussing, the target file format would need to be able to support the general Compass model. I'm not sure that leaves many options there are other than HDF4 and HDF5 (and I don't see much value in a HDF4 exporter). Would it be simpler to ditch the export-plugin concept and just have an HDF5 exporter?

— Reply to this email directly or view it on GitHubhttps://github.com/HDFGroup/hdf-compass/issues/118#issuecomment-150347136.

giumas commented 8 years ago

Attempting to summarize, a possible solution would be:

Did I miss/misunderstand something?

andrewcollette commented 8 years ago

Sounds good to me. I think having a small number of export plugins that work exclusively with the standard Array, Image, etc., interfaces would be nice. Using only those interfaces would (deliberately) limit how complicated they could get and keep the program streamlined.

jreadey commented 8 years ago

Would 'map' be a big confusing as a name? E.g.. confusion with the Python map.

Would Map derive from Array?

What about non-array based nodes (say equivalent to a Shapefile)?

giumas commented 8 years ago

Using only those interfaces would (deliberately) limit how complicated they could get and keep the program streamlined.

@andrewcollette For export as a geographic format (tiff, ESRI ascii grid, etc.), something like:

are required. This could become properties on the Map-derived class. With the BAG format, this information are provided in the XML metadata.

Would 'map' be a big confusing as a name? E.g.. confusion with the Python map.

@jreadey What do you suggest as name of this new node? MapArray? GeoArray? Chart?

Would Map derive from Array?

I don't see big differences in both solutions (deriving from Array or from Node). Do you see big advantages in a specific solution?

What about non-array based nodes (say equivalent to a Shapefile)?

If I well understood your point, the result would look a lot like ogr2ogr.. So I would leave it out for the moment. Maybe for the future development of a store-based export plugin mechanism.

ghost commented 8 years ago

suggest as name of this new node? MapArray? GeoArray? Chart?

How about: CoverageMap. Or is this too OGC-specific?

jreadey commented 8 years ago

I like GeoArray. Analogous to GeoTiff.

giumas commented 8 years ago

@jreadey About your size concerns:

In order to keep the size of a cartopy release low, the majority of data is not included as standard

from the official Cartopy's documentation: http://scitools.org.uk/cartopy/docs/latest/developer_interfaces.html#data-download-api

giumas commented 8 years ago

We might implement 2 types of geographic array frames:

giumas commented 8 years ago

Here a first implementation of what I mean:

untitled

untitled2

giumas commented 8 years ago

The test BAG file used for the shaded relief can be downloaded from: http://surveys.ngdc.noaa.gov/mgg/NOS/coast/W00001-W02000/W00275/BAG/W00275_MB_50cm_MLLW_1of1.bag

giumas commented 8 years ago

The visualization part is implemented by https://github.com/HDFGroup/hdf-compass/pull/150. The other ideas present on this ticket (e.g., the data-export functionality) does not fit so well with the current title. So they should become stand-alone ticket/PR when it is time to be implemented.