Browse XML, Download and other features not working

Hi there,

Allow me to introduce: I'm Tim-Hinnerk Heuer, a geo software developer at Landcare Research New Zealand. As far as I'm aware Landcare Research has an ongoing relationship with BRGM.

I was told that this plugin works and has quite a lot of advanced features. However, I was unable to get anything other than the GetCapabilities feature to work. For this I had to change this line: https://github.com/geekdenz/gml_application_schema_toolbox/blob/1135eb7bb8c17449f41521b578cde2583709bc8d/gml_application_schema_toolbox/gui/download_wfs2_panel.py#L52

Seems like I have a later version of the owslib library in Python 3 that may have changed their defaults.

I'm using a very recent version of QGIS: https://github.com/qgis/QGIS/tree/f579f1a449b0bf21902570a32dafc9546cb414bd or f579f1a

I cannot figure out how to get this plugin to work properly. I tried your WFS: https://wfspoc.brgm-rec.fr/geoserver/ows from the download tab.

I get this error:

2017-10-19T17:21:04 WARNING Traceback (most recent call last):
              File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 377, in _make_request
                httplib_response = conn.getresponse(buffering=True)
            TypeError: getresponse() got an unexpected keyword argument 'buffering'

            During handling of the above exception, another exception occurred:

            Traceback (most recent call last):
              File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 379, in _make_request
                httplib_response = conn.getresponse()
              File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
                response.begin()
              File "/usr/lib/python3.5/http/client.py", line 297, in begin
                version, status, reason = self._read_status()
              File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
                line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
              File "/usr/lib/python3.5/socket.py", line 575, in readinto
                return self._sock.recv_into(b)
              File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
                return self.read(nbytes, buffer)
              File "/usr/lib/python3.5/ssl.py", line 791, in read
                return self._sslobj.read(len, buffer)
              File "/usr/lib/python3.5/ssl.py", line 575, in read
                v = self._sslobj.read(len, buffer)
            socket.timeout: The read operation timed out

            During handling of the above exception, another exception occurred:

            Traceback (most recent call last):
              File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
                timeout=timeout
              File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 610, in urlopen
                _stacktrace=sys.exc_info()[2])
              File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 247, in increment
                raise six.reraise(type(error), error, _stacktrace)
              File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
                raise value
              File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
                body=body, headers=headers)
              File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
                self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
              File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
                raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
            requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='wfspoc.brgm-rec.fr', port=443): Read timed out. (read timeout=30)

            During handling of the above exception, another exception occurred:

            Traceback (most recent call last):
              File "/home/users/heuert/dev/cpp/QGIS/build-master/output/python/plugins/gml_application_schema_toolbox/gui/download_wfs2_panel.py", line 98, in on_downloadButton_clicked
                out = self.download()
              File "/home/users/heuert/dev/cpp/QGIS/build-master/output/python/plugins/gml_application_schema_toolbox/gui/download_wfs2_panel.py", line 155, in download
                response = wfs.getfeature(**params)
              File "/usr/local/lib/python3.5/dist-packages/owslib/feature/wfs200.py", line 221, in getfeature
                username=self.username, password=self.password)
              File "/usr/local/lib/python3.5/dist-packages/owslib/util.py", line 179, in openURL
                **rkwargs)
              File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
                return session.request(method=method, url=url, **kwargs)
              File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
                resp = self.send(prep, **send_kwargs)
              File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
                r = adapter.send(request, **kwargs)
              File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
                raise ReadTimeout(e, request=request)
            requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='wfspoc.brgm-rec.fr', port=443): Read timed out. (read timeout=30)

Is anything working in this plugin yet? Do I need specific versions of things?

owslib.__version__
'0.15.0'

We are considering contributing to this plugin a XML/WFS browse facility, which I thought was at least partially implemented.

Any help will be greatly appreciated.

Hi,

Thanks for your interest in the plugin.

I currently use QGIS 2.99.0-194 (installed under Windows through OsGeo4W 'Advanced install') and the plugin works fine. Code revision https://github.com/qgis/QGIS/commit/df0d717f92 compiled with GDAL/OGR 2.2.2

As far as the 3 tabs are concerned the status is the following

"download": GetCaps works fine using (https://wfspoc.brgm-rec.fr/geoserver/ows). There are glitches to be fixed such as : interacting with stored queries, fun with the usual x,y VS WFS version for the 'filter by extent'. We also want to link it better to the convert tab. Other wise from this one you generate a file which you have to point (again) in the 'convert' tab
"convert": is able to load either files or flows in XML or database mode (which uses GDAL GMLAS driver) Some datatypes trigger specific widgets (WaterML2 Part 1 - timeseries, INSPIRE PointTimeSeries, GWML2 GW_GeologyLogCoverage). For now, only in XML mode (planned to be ported in the Database mode) Both modes handle a primer of content negociation defining what to do when dereferencing a URI.
"export" It writes a GML file back (it's not WFS-T) after it has been imported in the tool. Example UseCase : the user receives a GML file and correct some elements into it (ex : label, URI to a vocab, ...) and export it back to the initial provider

Have you had the opportunity to look at some presentation we did on it to get an overview of the features from both the plugin and the GDAL GMLAS driver ? I just uploaded the presentation from the workshop we had at the latest FOSS4G-E: https://github.com/BRGM/gml_application_schema_toolbox/tree/master/presentations/2017_FOSS4G-E

Once you checked those, it would be good we exchange on what functionalities you plan to add and how we can coordinate things.

Best, Sylvain

Hi @sgrellet Thanks for getting back so quickly. However, I was unable to get the plugin working with QGIS 3

commit df0d717f928e0d8b7ceb7edfb69bee09266e29c7
Author: Nyall Dawson <nyall.dawson@gmail.com>
Date:   Wed Oct 18 10:51:43 2017 +1000

    Fix disabled add button in ogr source select for valid protocols

    Fixes #17206

I think it should work on Linux as well. Could you please look into fixing this? Maybe with the latest QGIS code would be good. I'm running Ubuntu 16.04.

Please let me know if you need more information.

Regards, Tim

@mhugo : any idea how to help Tim ?

Hi,

For this I had to change this line: https://github.com/geekdenz/gml_application_schema_toolbox/blob/1135eb7bb8c17449f41521b578cde2583709bc8d/gml_application_schema_toolbox/gui/download_wfs2_panel.py#L52

@geekdenz for what did you change it ?

I tried your WFS: https://wfspoc.brgm-rec.fr/geoserver/ows from the download tab.

It appears to be a timeout problem when trying to reach the brgm server. Is using this url in a browser working correctly ?

owslib.version '0.15.0'

I have a 0.10.0 version on Ubuntu 16.04 and 0.14.0 is installed with OSGEO4W64. No particular problem so far. How did you get this 0.15.0 version ? Would you have the opportunity to test the plugin under Linux ?

Hi @mhugo

I can test in Linux, no problem.

0.15.0 seems to be the default on Ubuntu 16.04 with Python 3. I could try downgrading or even fixing the problem. Maybe I could simply increase the timeout.

The reason I am looking at this plugin is that our company is interested in extending it and I've been given some time to do it. So, I need to know what features this plugin gives us out of the box first and then evaluating what we can contribute without duplicating effort.

I have developed a QGIS plugin before for QGIS 2.14, but with Python 2 and Qt 4. I am happy to learn, but to be honest I am not a QGIS expert user and have to do quite a bit of trial and error to find all the extensive menus etc. I have more experience with it than with ArcGIS or other commercial GIS. However, I've been working mainly with Web GIS for 7 years now and know my way around PostGIS, OpenLayers 2,3,4, was active at MapFish Print for a while, geospacial mathematics, i.e. vector mathematics etc well, but am not a WFS expert user.

I am happy to take this conversation to the mailing list or similar as a Github issue might not be completely appropriate. You could send me a private message on Twitter @geekdenz or here on Github.

But for now, I am happy if I can just get this plugin working somehow. If it works on Windows and since it is Python it should be reasonably easy to get it working in Linux as well.

The timeout may be, because I am in New Zealand and the connection may not be quick enough although we have a 1Gbit fibre connection. The crashes shouldn't happen though if we have the same version of things. We have some WFS 2.0.0 services with Snowflake that I could use internally, but there also, nothing was working for me.

We want to extend the plugin to provide an XML browse facility beyond a simple tree structure like in the "Show capabilities". The idea is to browse WFS and possibly extend to ontologies or Linked Data somehow. A geo-spacial browse facility for structured semi-arbitrary XML structures is what I understand my managers want. I understood there to be a WFS 2 plugin already, but couldn't find it in QGIS.

Any help and collaboration appreciated.

OK, I was finally able to get some things to work.

Had to comment out this line: https://github.com/geekdenz/gml_application_schema_toolbox/blob/a4b7607981f7515cb1ad890ebc2d997a4a3bee63/gml_application_schema_toolbox/gui/dockwidget.py#L56

I'm using https://github.com/qgis/QGIS/commit/f579f1a

Your presentations at https://github.com/BRGM/gml_application_schema_toolbox/tree/master/presentations/2017_FOSS4G-E seem to suggest that there are more Windows to this plugin or at least I was unable to find them.

Which WFS endpoints were you using in the workshop?

Hi,

"The idea is to browse WFS and possibly extend to ontologies or Linked Data somehow."

This highly resembles something I have in my roadmap as well (ex : consuming JSON-LD using the plugin). It may be good to continue that discussion via e-mail. You can contact me using s.grellet@brgm.fr

"I understood there to be a WFS 2 plugin already"

Actually, the download tab was somehow a port&adapt of the WFS2 plugin that worked under QGIS2. Given that I mainly focus on the 'convert' Tab (see below), there may be things pending to adjust as I wrote last week.

"Which WFS endpoints were you using in the workshop?"

I mainly use the convert Tab most of the time attacking a single Feature instance. Reusing the boxes from slide 3 of '20170718_QGIS_GMLAS_toolbox_groundwater_monitoring_BRGM_OSLANDIA.pptx', you can try loading :

-- 1°/ Borehole : start with this one : 'https://forge.brgm.fr/svnrepository/epos/trunk/instances/BoreholeView.xml' you will be able to dereference URI to

vocabulary endpoints : http://inspire.ec.europa.eu/codelist/.... or http://resource.europe-geology.eu/vocabs/...
other features for example A/ gsmlp:geologicalDescription will provide you with a GWML2 GW_GeologyLogCoverage geological log (should trigger a borehole log viewer in XML mode) B1/ gsmlp:groundWaterLevel will provide you with a description of the Piezometer attached to the Borehole according to INSPIRE Environmental Monitoring Facilities data specification. B2/ From that flow, you'll be able to dereference content from SOS endpoints. Try the ef:hasObservation asking for WaterML 2.0 and you'll receive raw groundWaterLevel observation acquired at that piezometer (be a patient it's a 2 year long time series with a measure per hour). This will trigger a timeseries viewer in XML mode.

-- 2°/ Environmental Monitoring Facility -> GWML2:Aquifer using for example that piezometer ( http://ressource.brgm-rec.fr/data/Piezometre/02713X0132/PZ-8.2 ) you will be able to dereference

the GWML2 description of the aquifer it is monitoring using the content from ef:featureOfInterest
and also (as before) the link to groundWaterLevel timeseries available from it

HTH Sylvain

BRGM / gml_application_schema_toolbox

Browse XML, Download and other features not working #34