wigle_api_lite.py might be broken?

[richfairhurst]

From snoopy.log:

2013-01-09 13:56:31 INFO pytail.py: New probe data! 2013-01-09 13:56:34 INFO ssid_to_loc.py: Looking up address for 8 SSIDs 2013-01-09 13:56:34 INFO wigle_api_lite.py: Wigling xxxxxxx 2013-01-09 13:56:34 INFO wigle_api_lite.py: Using Wigle account xxxxxxxxx 2013-01-09 13:56:34 INFO connectionpool.py: Starting new HTTPS connection (1): wigle.net 2013-01-09 13:56:35 INFO ssid_to_loc.py: An error occured, will retry in 60 seconds (Unable to login to wigle) 2013-01-09 13:56:39 INFO prox_guid.py: 39 devices probing. Grouping into proximity sessions...

I'm clearly no python guru, but if I pull out the code from wiggle_api_lite and try and just isolate it, my understanding is that the below should show the contents of the cookie but fails to - which suggests (to my amateur mind :-) that wigle_api_lite.py is going to always fail as its not going to be able to read 'auth'

rich@h211:~# python

import requests    url={'login': "https://wigle.net/gps/gps/main/login", 'query':"http://wigle.net/gps/gps/main/confirmquery/"}  payload={'credential_0':'loginname', 'credential_1':'password'}  r = requests.post(url['login'],data=payload,timeout=10)  r.cookies['auth'] Traceback (most recent call last):   File "", line 1, in   File "/usr/local/lib/python2.7/dist-packages/requests/cookies.py", line 243, in getitem     return self._find_no_duplicates(name)   File "/usr/local/lib/python2.7/dist-packages/requests/cookies.py", line 284, in _find_no_duplicates     raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path)) KeyError: "name='auth', domain=None, path=None"  

Curl seems to work:

rich@h211:/tmp# curl --data "credential_0=username&credential_1=password" https://wigle.net/gps/gps/mail/login --dump-header headers

rich@h211:/tmp# curl -L -b headers --data "addresscode=&statecode=&zipcode=&variance=&latrange1=&latrange2=&longrange1=&longrange2=&lastupdt=&netid=&ssid=Hotspot&Query=Query" https://wigle.net/gps/gps/main/confirmquery

[sensepost]

Hi Rich,

Thanks for the feedback. You should be able to run the wigle_api_lite.py by itself, like so:

$ python wigle_api_lite.py BTHomeHub2-GQST 2013-01-11 11:13:44 INFO wigle_api_lite.py: Wigling BTHomeHub2-GQST 2013-01-11 11:13:44 INFO wigle_api_lite.py: Using Wigle account xxxxxxxx 2013-01-11 11:13:44 DEBUG wigle_api_lite.py: [+] Logging into wigle with xxxxxxxx:xxxxx via proxy '' 2013-01-11 11:13:44 INFO connectionpool.py: Starting new HTTPS connection (1): wigle.net 2013-01-11 11:13:45 DEBUG connectionpool.py: "POST /gps/gps/main/login HTTP/1.1" 302 261 2013-01-11 11:13:45 DEBUG connectionpool.py: "GET / HTTP/1.1" 200 4667 2013-01-11 11:13:46 DEBUG wigle_api_lite.py: [-] Successfully logged in with credentials xxxxxxxx:xxxxx via . 2013-01-11 11:13:46 DEBUG wigle_api_lite.py: [-] Looking up BTHomeHub2-GQST (xxxxxxxx ) 2013-01-11 11:13:46 INFO connectionpool.py: Starting new HTTP connection (1): wigle.net 2013-01-11 11:13:46 DEBUG connectionpool.py: "POST /gps/gps/main/confirmquery/ HTTP/1.1" 200 1716 2013-01-11 11:13:46 INFO wigle_api_lite.py: Saving streetview to /home/woodstock/snoopy/server/web_data/street_views/51.66931152,-1.76962757.jpg [{'ssid': 'BTHomeHub2-GQST', 'long': u'-1.76962757', 'last_update': u'20110714145521', 'mac': u'00:d0:44:bc:60:da', 'ga': {'g_status': '200', 'country': u'UK', 'code': u'GB', 'http_status': 200, 'address': u'High Street, Fairford, Gloucestershire GL7, UK'}, 'lat': u'51.66931152', 'overflow': 0, 'last_seen': u'2011-07-14 14:55:12'}]

Could you check if that works for you? Perhaps there's a typo in your credentials (you can manually check wigle_creds.txt)?

Glenn

[richfairhurst]

Desperately trying (hoping!) not to have made a dumb mistake here (i've been know to make the odd idiot of myself) but since I think I have run out of things to test, I'm going to chance my luck and post again :-)

I know a copy and paste from wigle_creds.txt works via the browser (and via curl etc) - and I've even reset password to be password to rule out any issues with special characters etc - Same issue if I grab the full wigle_api.py from the branch (using wigler_conf.cfg to hold the credentials)

If I run by itself I get similar errors:

root@h211:/tmp# ./wigle_api_lite.py BTHomeHub2-GQST 2013-01-11 14:58:37 INFO wigle_api_lite.py: Wigling BTHomeHub2-GQST 2013-01-11 14:58:37 INFO wigle_api_lite.py: Using Wigle account xxxxxx 2013-01-11 14:58:37 DEBUG wigle_api_lite.py: [+] Logging into wigle with xxxxx:xxxx via proxy '' 2013-01-11 14:58:37 INFO connectionpool.py: Starting new HTTPS connection (1): wigle.net 2013-01-11 14:58:38 DEBUG connectionpool.py: "POST /gps/gps/main/login HTTP/1.1" 302 261 2013-01-11 14:58:38 DEBUG connectionpool.py: "GET / HTTP/1.1" 200 4733 2013-01-11 14:58:38 DEBUG wigle_api_lite.py: [-] Error logging in with credentials xxxx:xxxx. Thread returning. {'error': 'Unable to login to wigle'}

I wondered if it was an access issue, so I throw up another VPS, ran as root just in case I broke some access issues, cleared iptables, and also double-checked on a local Ubuntu desktop as well.

If I run directly in python "r.cookie['auth']" returns an error, when I believe it should return the cookie, which is what I suspect is casing the failure to login. Seems to be working OK for you though - before I drive myself crazy, your using the same version of wigle_api_lite.py that I pulled from here?

Anyway it's Friday so I'm going home to drown my sorrows in a good whisky :-)

[sensepost]

Hey Rich,

Odd indeed. Here's some output which can perhaps help us debug this little gremlin:

wget https://raw.github.com/sensepost/Snoopy/master/snoopy/server/bin/wigle_api_lite.py

md5sum wigle_api_lite.py

69338e95c00804aab55efd4eb084728a wigle_api_lite.py

echo someuser:somepassword: > wigle_creds.txt

python -V

Python 2.7.3

python wigle_api_lite.py BTHomeHub2-GQST

2013-01-14 11:02:01 INFO wigle_api_lite.py: Wigling BTHomeHub2-GQST 2013-01-14 11:02:01 INFO wigle_api_lite.py: Using Wigle account xxxxxx 2013-01-14 11:02:01 DEBUG wigle_api_lite.py: [+] Logging into wigle with xxxxxx:xxxxxx via proxy '' 2013-01-14 11:02:01 INFO connectionpool.py: Starting new HTTPS connection (1): wigle.net 2013-01-14 11:02:01 DEBUG connectionpool.py: "POST /gps/gps/main/login HTTP/1.1" 302 261 2013-01-14 11:02:02 DEBUG connectionpool.py: "GET / HTTP/1.1" 200 4733 2013-01-14 11:02:02 DEBUG wigle_api_lite.py: [-] Successfully logged in with credentials xxxxxx:xxxxxx via . 2013-01-14 11:02:02 DEBUG wigle_api_lite.py: [-] Looking up BTHomeHub2-GQST (xxxxxx ) 2013-01-14 11:02:02 INFO connectionpool.py: Starting new HTTP connection (1): wigle.net 2013-01-14 11:02:02 DEBUG connectionpool.py: "POST /gps/gps/main/confirmquery/ HTTP/1.1" 200 1716 2013-01-14 11:02:02 INFO wigle_api_lite.py: Saving streetview to /web_data/street_views/51.66931152,-1.76962757.jpg [{'ssid': 'BTHomeHub2-GQST', 'long': u'-1.76962757', 'last_update': u'20110714145521', 'mac': u'00:d0:44:bc:60:da', 'ga': {'g_status': '200', 'country': u'UK', 'code': u'GB', 'http_status': 200, 'address': u'High Street, Fairford, Gloucestershire GL7, UK'}, 'lat': u'51.66931152', 'overflow': 0, 'last_seen': u'2011-07-14 14:55:12'}]

It's odd that your cookie doesn't get populated with the auth parameter. Could you manually verify what cookie attributes get set? Here's the output from the snippet of code you originally pasted:

import requests loginname="xxxx" password="xxxx"

... url={'login': "https://wigle.net/gps/gps/main/login", 'query':"http://wigle.net/gps/gps/main/confirmquery/"} payload={'credential_0':loginname, 'credential_1':password} r = requests.post(url['login'],data=payload,timeout=10) r.cookies['auth'] 'username%blahblahblahauthcookie' r.cookies <<class 'requests.cookies.RequestsCookieJar'>[Cookie(version=0, name='auth', value='username%blahblahblahauthcookie', port=None, port_specified=False, domain='.wigle.net', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=False, expires=1358168519, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False)]>

Glenn

[richfairhurst]

I put up a brand new ubuntu vm just in case i missed something / package on my vps host, ran the install.sh script and checked no errors and that I had all the needed packages etc and ran through the above:

rich@ubuntu:~$ cd /tmp rich@ubuntu:/tmp$ wget https://raw.github.com/sensepost/Snoopy/master/snoopy/server/bin/wigle_api_lite.py rich@ubuntu:/tmp$ md5sum wigle_api_lite.py 69338e95c00804aab55efd4eb084728a wigle_api_lite.py rich@ubuntu:/tmp$ echo xxxxxxxx:xxxxxxxxxx: > wigle_creds.txt rich@ubuntu:/tmp$ python -V Python 2.7.3 rich@ubuntu:/tmp$ sudo python wigle_api_lite.py BTHomeHub2-GQST [sudo] password for rich: 2013-01-14 17:09:14 INFO wigle_api_lite.py: Wigling BTHomeHub2-GQST 2013-01-14 17:09:14 INFO wigle_api_lite.py: Using Wigle account xxxxxxxxxx 2013-01-14 17:09:14 DEBUG wigle_api_lite.py: [+] Logging into wigle with xxxxxxxxxx:xxxxxxxxxxxx via proxy '' 2013-01-14 17:09:14 INFO connectionpool.py: Starting new HTTPS connection (1): wigle.net 2013-01-14 17:09:21 DEBUG connectionpool.py: "POST /gps/gps/main/login HTTP/1.1" 302 261 2013-01-14 17:09:27 DEBUG connectionpool.py: "GET / HTTP/1.1" 200 4735 2013-01-14 17:09:27 DEBUG wigle_api_lite.py: [-] Error logging in with credentials xxxxxxxxxx:xxxxxxxxxxxx. Thread returning. {'error': 'Unable to login to wigle'} rich@ubuntu:/tmp$ python Python 2.7.3 (default, Aug 1 2012, 05:16:07) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information.

import requests loginname="xxxxxxxxxx" password="xxxxxxxxxxxx" url={'login': "https://wigle.net/gps/gps/maib/login", 'query':"http://wigle.net/gps/gps/main/confirmquery/"} payload={'credential_0':loginname, 'credential_1':password} r=requests.post(url['login'],data=payload,timeout=10) r.cookies['auth'] r.cookies['auth'] Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/requests/cookies.py", line 243, in getitem return self._find_no_duplicates(name) File "/usr/local/lib/python2.7/dist-packages/requests/cookies.py", line 284, in _find_no_duplicates raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path)) KeyError: "name='auth', domain=None, path=None"

Double checked the wigle account details by cut and pasting from the terminal window of the above log into a browser session as was able to login ok.

Thought I might also post versions as well just in case:

rich@ubuntu:~/Snoopy$ dpkg -l ii python-pip 1.0-1build1 ii gcc 4:4.6.3-1ubuntu5 ii libxml2-dev 2.7.8.dfsg-5.1ubuntu4.3 ii libxslt1-dev 1.1.26-8ubuntu1.2 ii python2.7-dev 2.7.3-0ubuntu3.1 ii python-beaker 1.5.4-4+squeeze1build0.12.04.1 ii python-flask 0.8-1 ii python-jinja2 2.6-1 ii python-xapian 1.2.8-1 ii python-sqlalchemy 0.7.4-1ubuntu0.1 ii python-mysqldb 1.2.3-1build1

rich@ubuntu:~/Snoopy$ pip freeze

lxml==3.1beta1 BeautifulSoup==3.2.1 requests==1.1.0 PIL==1.1.7 mitmproxy==0.8.1 ipaddr==2.1.10 publicsuffix==1.0.4 Twisted==12.3.0 cryptacular==1.4.1 Flask-SQLAlchemy==0.16 SQLAlchemy==0.7.4

[sensepost]

Hi Rich,

Sorry for the delay - I've been on the road for work. So, it seems Requests version 1.1.0 may have a bug in the way it handles cookies and redirects. A quick fix may be to downgrade to a previous version, whilst I try get the gremlins out of 1.1.0. Let me know if that works for you?

Glenn

[richfairhurst]

ditto to being on the road for work :-) I downgraded to requests 0.14.0 and got it working and I'm now seeing wigle requests returning good information.