search

geopython / MetaSearch

*** We have moved! MetaSearch is now a core QGIS plugin! (2014-11-30). See QGIS repository
https://hub.qgis.org/wiki/quantum-gis/MetaSearch
Other
4 stars 12 forks source link

add "add Data" functionality #25

Closed tomkralidis closed 9 years ago

tomkralidis commented 10 years ago

Allow the user to download data from the record (like a Shapefile, GeoTIFF, etc.) and load directly into the map canvas. See https://github.com/geopython/MetaSearch/wiki/Discovery-workflow for more info.

ggiuliani commented 10 years ago

@tomkralidis I think the workflow you're proposing is fine. From my point of view, it wouldn't be too difficult to identify whether data to download are in vector or raster formats. I think it is similar with what is already implemented in the plugin. If in the metadata record, there is a WFS endpoint, it is a vector. Same of raster with WCS. Then the client must handle the URL to download the data:

Regarding where to download the data, can we imagine a temporary folder (maybe there is a default one in QGIS)?

rduivenvoorde commented 10 years ago

@astroimagine IF the data is served as wms/wfs/wcs service, it is already handled by QGIS/MetaSearch. A wfs indeed retrieves vectordata, and a wcs retrieves a coverage (as geotiff or whatever the server serves).

regarding a temporary folder: Qt has it's standard folders (for different platforms): http://doc.qt.digia.com/4.6/qdesktopservices.html#StandardLocation-enum

Have a look at what your cws is serving as types of 'links' by doubleclicking on one of the results an look in the Links block.

There are a whole set of datadownload possibilities (of which we already can handle the OGS:xxx types). For most of the possibilities, see: https://github.com/geopython/pycsw/wiki/Geonode-notes#wiki-protocol-enumeration-and-description

For example these links: WWW:DOWNLOAD-1.0-ftp--download File for download through FTP WWW:DOWNLOAD-1.0-http--download File for download FILE:GEO GIS file FILE:RASTER GIS RASTER file

seem doable, but I haven't seen them yet in our national csw, where we often serve atom.xml as type:

screenshot from 2014-02-26 12 08 11

So the crux is to transparantly and without errors handle all the different options

The link in the image above links to: http://geodata.nationaalgeoregister.nl/brpgewaspercelen/atom/brpgewaspercelen.xml which we could automatically handle, but we could also style it and show it in the same box (as your browser shows it). Then the user can theirself download/save it and open it in QGIS

ggiuliani commented 10 years ago

@rduivenvoorde thanks a lot for your comment. Maybe I was not that much clear in my previous message. I was aware of the QGIS/Metasearch capabilities of adding WMS/WFS/WCS in the list of related services. But from a end-user perspective (e.g., a GIS analyst), I found a bit "frustrating" to discover a metadata record with for example a WFS (e.g, endpoint + layer name) and when I click on the "Add WFS" this adds the endpoint to the WFS service list. Then I need to browse the entire list of layers served by this WFS to try to find the layer I'm interested in. From my point of view a metadata record describes a layer that can have a WxS endpoint with a layer name. From that it would be great to directly download the data in the Map Canvas (no need to browse the entire list of layers available in a WxS instance). Otherwise an alternative is at least to have the layer selected in the list of layer served by the instance so that we can immediately identify the layer related to the metadata record. Is it possible?

rduivenvoorde commented 10 years ago

a clear. Yes, I understand what you want to achieve now.

But I think we face a standardization problem here (for what I've heard, I'm not a csw standard guru): I'm told that there is no strict way defined how to define a layer name for a OWS service. So some users define it as part of the capabiliies url, or define both a capabilities and a wms/wfs url. While others (as in our national one) use the title of the service for the layername to be used.

So we could define some heuristic to determine this, but.... it would be easier if there was some common way to define this. @tomkralidis I've seen some discussion page about harmonisation between different foss csw implemenations, but was not able to find it anymore...

Besides this, it helps if the WMS administrators do not define those huge global service lists anymore. Eg geoserver has the notion of creating 'virtual OWS services': http://docs.geoserver.org/latest/en/user/services/virtual-services.html With mapserver it is even easier to achieve. But still there is then the 'choose layer' dialog in between, but at least you do not have to scroll to the 45th layer :-) Another thing to note: there is also not concensus what exactly is the relation between a wms layer and a csw record: some create a record for every layer, while others create a record for a group or set of layers. Though for wfs it is indeed more logical that one service corresponds to one csw record.

ggiuliani commented 10 years ago

@rduivenvoorde thanks a lot for this very interesting explanation. You're right we are really facing standardisation problem here. What we will probably do, in our office, is to develop some best practices documentation in order to ensure that collaborators will prepare data and metadata according some "internal" rules :-)

tomkralidis commented 10 years ago

cc @bartvde

Agreed, the problem here is the process of adding data from a service. MetaSearch (currently) supports OGC CSW CORE (Dublin Core). There's no pure way to achieve this (though GeoNetwork uses dc:URI, which is not implemented by other CSWs) using DC. Some work is going on at https://github.com/geopython/pycsw/wiki/Geonode-notes#wiki-metadata-links to help this.

The CSW ISO profile allows you to do this, but it means making two queries to the CSW. I've cc'd @bartvde who may have comments based on his extensive adventures in this area :)

bartvde commented 10 years ago

Right ISO AP normally requires 2 requests if I have understood things correctly (and used by INSPIRE).

Though the Dutch metadata profile has a way in ISO to embed this in the metadata record directly, but I've forgotten the details. I'm quite sure @thijsbrentjens can remind us of the details here, he is the real expert (not me ;-) ).

tomkralidis commented 10 years ago

@astroimagine do you have any metadata records in your CSW that have direct links to data? Like downloading a Shapefile, etc.?

thijsbrentjens commented 10 years ago

thanks @bartvde for not putting any pressure on me... I'll try to clarify what is done in the Netherlands. My knowledge of the profiles is getting a bit rusty, but indeed, in the Netherlands we do use some elements (additional to INSPIRE metadata elements) in dataset metadata to make it possible to directly access a specific layer (for WMS) or featuretype (for WFS), without an extra step. In NL we use the gmd:name element in MD_DigitalTransferOptions to refer to a WMS Layer or WFS FeatureType.

You could get these elements by one GetRecords request (assuming this is what MetaSearch uses now) and the outputschema for ISO metadata. For example with this GetRecords request: http://nationaalgeoregister.nl/geonetwork/srv/dut/inspire?&request=GetRecords&service=CSW&version=2.0.2&resultType=results&constraintLanguage=FILTER&constraint_language_version=1.1.0&typeNames=csw%3ARecord&elementSetName=full&OutputSchema=http%3A//www.isotc211.org/2005/gmd&maxRecords=9

And look for the second record, pointing to a WFS with URL http://agentschapnl.kaartenbalie.nl/cgi-bin/mapserv?map=/srv/maps/agentschapnl/RVO_WarmteAtlas.map&version=1.0.0.

This element gmd:name for this usage is only mandatory for dataset metadata in the Netherlands though and not by definition used in service metadata. Which is a bit confusing I can imagine.

Hope this helps.

ggiuliani commented 10 years ago

@tomkralidis not yet but one of my colleagues is currently preparing metadata with link to zip files that contain shapefiles. I'll let you know as soon as it is ready.

rduivenvoorde commented 10 years ago

one idea that I now want to try to build:

tomkralidis commented 9 years ago

@astroimagine do you have any GeoTIFF and / or Shapefile zip downloads in any metadata on CSWs we can test? I'd like to implement this for 0.4.0 release.

ggiuliani commented 9 years ago

@tomkralidis Yes we have some resources :-) Shapefile in a zip: http://metadata.grid.unep.ch:8080/geonetwork/srv/eng/metadata.show?id=1882&currTab=simple GeoTiff (served via WCS): http://twapgeoportal.grid.unep.ch:8080/geonetwork/srv/eng/metadata.show?id=1112&currTab=simple

Greg

On 02 Sep 2014, at 17:45, Tom Kralidis notifications@github.com wrote:

@astroimagine do you have any GeoTIFF and / or Shapefile zip downloads in any metadata on CSWs we can test? I'd like to implement this for 0.4.0 release.

— Reply to this email directly or view it on GitHub.

Dr. Gregory Giuliani Scientific collaborator - Spatial Data Infrastructure http://www.unige.ch/envirospace/People/giuliani.html

University of Geneva Institute for Environmental Sciences/enviroSPACE 7 route de Drize, CH - 1227 CAROUGE Phone: (+41 22) 379 07 09 Fax: (+41 22) 379 07 44 Web: http://www.unige.ch/envirospace Email: gregory.giuliani@unige.ch

UNEP/DEWA/GRID-Geneva International Environment House 11, chemin des Anémones, CH - 1219 CHATELAINE Phone: (+41 22) 917 84 17 Fax: (+41 22) 917 80 29 Web: http://www.grid.unep.ch Email: gregory.giuliani@unepgrid.ch

ninsbl commented 9 years ago

Hi Greg, I am just following the discussion as I would be interested in an "add data - feature" as well. However, the links within both metadata-etities (not the links to the metadata) do not seem to work. In both cases I get "The webpage cannot be found" error when I try to access the data. For the Geotiff it should be probably: ftp://datastorage.grid.unep.ch/world/population_distribution_demography/bc9090_gnvgnv029_world_cities_population.zip, no idea for the shape file... Stefan

ggiuliani commented 9 years ago

Hi Stefan,

Thanks a lot for your feedback. That’s strange because I’ve tested the ftp link and I’m able to download the zip file. Apparently, this is not a network issue because I can successfully download the data from home.

Cheers

Greg

On 03 Sep 2014, at 11:18, ninsbl notifications@github.com wrote:

Hi Greg, I am just following the discussion as I would be interested in an "add data - feature" as well. However, the links within both metadata-etities (not the links to the metadata) do not seem to work. In both cases I get "The webpage cannot be found" error when I try to access the data. For the Geotiff it should be probably: ftp://datastorage.grid.unep.ch/world/population_distribution_demography/bc9090_gnvgnv029_world_cities_population.zip, no idea for the shape file... Stefan

— Reply to this email directly or view it on GitHub.

Dr. Gregory Giuliani Scientific collaborator - Spatial Data Infrastructure http://www.unige.ch/envirospace/People/giuliani.html

University of Geneva Institute for Environmental Sciences/enviroSPACE 7 route de Drize, CH - 1227 CAROUGE Phone: (+41 22) 379 07 09 Fax: (+41 22) 379 07 44 Web: http://www.unige.ch/envirospace Email: gregory.giuliani@unige.ch

UNEP/DEWA/GRID-Geneva International Environment House 11, chemin des Anémones, CH - 1219 CHATELAINE Phone: (+41 22) 917 84 17 Fax: (+41 22) 917 80 29 Web: http://www.grid.unep.ch Email: gregory.giuliani@unepgrid.ch

tomkralidis commented 9 years ago

As per #62 moved to http://hub.qgis.org/issues/11733