Issues accessing NBN data behind proxy

[mtravis]

I'm trying to access data using the NBN tool but unfortunately getting a proxy error.

e = {URLError} URLError(error('Tunnel connection failed: 407 Proxy Authorization Required',),)

I can incidently get through to the main site and retrieve data via a browser using: https://data.nbn.org.uk/api/taxa?q=robin

I have added my proxy settings to the network tab and can access the plugins repo, etc. Have you set this up to work behind a proxy?

Thanks

Matt

[burkmarr]

Hello Matt,

It's something I didn't even think about. I'm surprised that I've not had questions like yours before.

I guess that the way it should work is that the plugin should check the QGIS network proxy settings to see if they are specified and, if they are, then use them in the calls to the NBN web services. At the moment it doesn't do this. I found this thread which I think gives some clues about how it should work: http://stackoverflow.com/questions/23041066/setting-proxy-parameter-in-qgis-plugins-how-to

I'll take a look over the next couple of weeks and see if I can fix it.

Rich

[mtravis]

Great. I found that link too and have been doing a bit of delving into proxy settings in python. I can get to the proxy settings using the script below but not sure how to pass them to urllib2 as per this method: http://www.learntosolveit.com/python/web_urllib2_proxy_auth.html (I've even hardcoded my proxy details in and that doesn't seem to work either so maybe its a setting on our firewall).

from PyQt4.QtNetwork import QNetworkRequest, QNetworkProxy
from qgis.core import QgsNetworkAccessManager
import urllib2

s = QSettings() #getting proxy from qgis options settings
proxyEnabled = s.value("proxy/proxyEnabled", "")
proxyType = s.value("proxy/proxyType", "" )
proxyHost = s.value("proxy/proxyHost", "" )
proxyPort = s.value("proxy/proxyPort", "" )
proxyUser = s.value("proxy/proxyUser", "" )
proxyPassword = s.value("proxy/proxyPassword", "" )
if proxyEnabled == "true": # test if there are proxy settings
   proxy = QNetworkProxy()
   if proxyType == "DefaultProxy":
    proxy.setType(QNetworkProxy.DefaultProxy)
   elif proxyType == "Socks5Proxy":
    proxy.setType(QNetworkProxy.Socks5Proxy)
   elif proxyType == "HttpProxy":
    proxy.setType(QNetworkProxy.HttpProxy)
   elif proxyType == "HttpCachingProxy":
    proxy.setType(QNetworkProxy.HttpCachingProxy)
   elif proxyType == "FtpCachingProxy":
    proxy.setType(QNetworkProxy.FtpCachingProxy)
   proxy.setHostName(proxyHost)
   proxy.setPort(int(proxyPort))
   proxy.setUser(proxyUser)
   proxy.setPassword(proxyPassword)
   QNetworkProxy.setApplicationProxy(proxy)
   print proxyHost

[burkmarr]

I've done a lot of work on this over the weekend Matt and I had hoped to have had a release ready today, but sadly the NBN web services have been down since last night (as often happens at weekends) and I'd be surprised if they were revived before tomorrow - making further work and testing of my fix for this impossible.

The solution is built around the QGIS supplied QgsNetworkAccessManager object. If you have proxy settings set in the QGIS options, then this object works with those options - so you don't have to manipulate proxy settings directly with lower level QT stuff (as in your previous post).

I'm almost there with it - just one problem to sort out with timeouts on long calls, e.g. for downloading data. I've been testing it by setting up a proxy server on my machine with the Fiddler program and setting my Windows firewall options to disallow any access to the Server except through the Fiddler proxy server. So I think I'm working in a realistic test environment, but it will be good to get you to try it out. Would you be willing to try out a beta release before I publish it?

Final thing to note - the current NBN web services are being 'switched off' at the end of March when the NBN moves to a new 'NBN Atlas' architecture. They are replacing the web services, but they have yet to publish the test versions of replacement services - or even their APIs - so there's little that third-party developers can do in advance to prepare for the switch-over. The plugin's NBN tool will have to be completely overhauled when the new services are published. Nevertheless, this work on using the QgsNetworkAccessManager object will not be in vain since the same issues of working behind a proxy will apply for the new web services.

Best wishes,

Rich

[mtravis]

Hi Rich

Thanks for the update. Definitely happy to help test out. I've actually managed to get the NBN working with our proxy by adding some exception rules.

I had heard that the NBN was changing to the Atlas of England. Hopefully they won't change the API too much.

Thanks again for your work on this and the great plugin.

Cheers

Matt

[burkmarr]

Okay great. If you want to try it out, goto this repository - https://github.com/burkmarr/QGIS-Biological-Recording-Tools/tree/ProxyServer- and click the 'Clone or download' button and select 'Download zip'.

When you unzip the file you will have a folder called 'QGIS-Biological-Recording-Tools-ProxyServer'. Inside this you will have another folder of the same name. Copy this second folder (i.e. the inner one) and paste it to the QGIS plugins folder on your computer. (On my computer this is 'C:\Users\fsc.qgis2\python\plugins'.)

Now when you open QGIS and go to the Plugin manager, uncheck the box next to your currently installed version of the plugin. Goto the uninstalled plugins and you should see a couple with the name 'TomBio tools'. If you highlight each of them and look in the information panel on the right, you want the one that says something like 'Installed version: 2.6.0 (in C:\Users\fsc.qgis2\python\plugins\QGIS-Biological-Recording-Tools-ProxyServer)' - look for the 'ProxyServer' text. That's the one you want to activate - so click the checkbox next to it.

If you've changed your firewall settings to make an exception for QGIS or whatever, then you will need to reset those to test the new functionality properly! Set your proxy servers settings in the QGIS network settings (also increase the timeout value if you are planning to test the download feature).

Let me know how you get on.

Best wishes.

Rich

[burkmarr]

Release 2.6.1 released.