TypeError: 'NoneType' object is not iterable

[sagilo]

I've install version 3.0.6 inside a virtual environment Running test.py:

service = Service(username, password, 'google_maps_location_sharing.conf')
for person in service.get_all_people():
    print(person)

Output:

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    for person in service.get_all_people():
  File "/home/sagi/venv/temp/lib/python3.5/site-packages/locationsharinglib/locationsharinglib.py", line 459, in get_all_people
    people = self.get_shared_people() + [self.get_authenticated_person()]
  File "/home/sagi/venv/temp/lib/python3.5/site-packages/locationsharinglib/locationsharinglib.py", line 420, in get_shared_people
    for info in output[0]:
TypeError: 'NoneType' object is not iterable

Username & password belongs to a new google account which I've shared my location with. I've logged in using Firefox to the new account before running the program.

[redeye86]

Hi,

i also created a new dedicated account which shows the location of the other users and i get the same error. I inserted a print(response.text) in line 404. Returns:

)]}'
[null,null,"XXXXXXXX-XXXXXXXXXXX-XXXXXXXXXXXXXX","XXXX_XXXXXXXXXXXXX",null,null,"XXXXXXXXX",1800,1539463534567]

System is Kubuntu 18.04; environment set up using:

virtualenv --python=python3.6 env
cd env/
./bin/pip install locationsharinglib
./bin/python3 test.py

test.py contents:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

username="XXXXXX@gmail.com"
password="XXXXXXXXXXXXXXXXXXX"
cookies_file="cookie.txt"

from locationsharinglib import Service
service = Service(username, password, cookies_file)
#for person in service.get_all_people():
for person in service.get_shared_people():
    print(person)

[costastf]

The returned value is not a valid location value.there are many fields missing, that is why it craps out. In a valid response the location data exists in the first index of a returned array. I can handle this issue better, but since you are not getting valid data, you will not have location info, just an empty list.

[costastf]

If you access this account through the web interface, do you get your shared locations? If so then it would be nice if you could look into the response that you get in case it is a different version of the api that I do not handle at all.

[redeye86]

Yes, i can see the location in the web interface. I recorded a request in Chrome-Dev-Tools and exported it. I anonymized and uploaded it as a gist:

HAR: https://gist.github.com/redeye86/88df3fcb53a5df226cc950b3e982cdb2

Response: https://gist.github.com/redeye86/a0258bd6616ea6dedaac9a65c3a652ff

Hope that helps.

BTW:

If i run ./bin/python cli.py --email XXXXXX@gmail.com --password XXXXXXXXX i get some html output. I guess it is the captcha formular that is discussed in the other threads.

[costastf]

That response is fine. I guess you got the captcha, that is why you don't get a proper response.I will extend the logging and try to figure things out tomorrow.

[lufton]

Yeap, the same behavior. I rendered response.text from _submit_password and found capcha container under password field.

[lufton]

After debugging I figured out that there should be four required cookies set to get valid response: SID, HSID, SSID and SIDCC.

[costastf]

If there is a captcha then the authentication will not succeed, so the response will not be complete. The only thing that can be done is the library blowing up with an exception in case of a captcha so there is no confusion to the user.

[lufton]

I'm afraid this is not about captcha. It is sign in mechanism. As I use the same IP to sign manually and it never ask me to enter captcha. I'll try to debug deeper and see how it works. BTW, @costastf where did you get /signin/v1/lookup as manual sign in routine sends request to /_/signin/sl/lookup? Is this is legacy method?

[costastf]

I think Google uses different mechanisms for logging in depending your locale. So apparently where you are from it uses something different. I will extend the debugging to log all actions so we can identify all differences and try to implement them.

[elad-bar]

the problem is with the function that returns all people, it returns shared people + authenticated person: people = self.get_shared_people() + [self.get_authenticated_person()]

the problem with that is the authenticated person returns None at array index [13] which fails it, to fix it updated google_maps.py to use get_shared_people instead of get_all_people

[costastf]

Do you have a trace for this? None should be filtered out by the get_all_people method so it should not create an issue.

[lufton]

the problem is with the function that returns all people, it returns shared people + authenticated person: people = self.get_shared_people() + [self.get_authenticated_person()]

the problem with that is the authenticated person returns None at array index [13] which fails it, to fix it updated google_maps.py to use get_shared_people instead of get_all_people

I believe it is not a problem. The problem is the script didn't receive correct cookies as google changed signin routine.

[elad-bar]

I wrote it after doing the exact same thing after experiencing the same issue, without clearing the cookie, From the code as you can see in function get_authenticated_person, index 13 is None, that's the issue: person = Person([ self.email, output[9][1], None, None, None, None, [ None, None, self.email, self.email ], None, None, None, None, None, None, None, ])

[lufton]

For now I just add this is _get_data method:

        cookies = {
            'SID': 'mgYWbo6MJMEttBCFR9hDmGVvsIG5ocnflMN-KnL71qDTmSV72V_et3IcOgkIEuXBbXXXXXX',
            'HSID': 'A8GKG-d5SECXXXXXX',
            'SSID': 'APwo1LNb38KXXXXXX',
            'SIDCC': 'AGIhQKT4C_jfPf-Xk_7xZDdcZHyKwNttmFqE9yZqQA5r4ufFPfceqD8oDGXbte6Tk9XXXXXX'
        }
        response = self._session.get(url, params=payload, cookies=cookies)

I got those from chrome developer panel:

[redeye86]

I figured out, that SID, HSID and SSID are sufficient.

[lufton]

I figured out, that SID, HSID and SSID are sufficient.

Actually as I told:

After debugging I figured out that there should be four required cookies set to get valid response: SID, HSID, SSID and SIDCC.

[costastf]

Guys as I have already mentioned I think that Google uses different authentication methods per location. So in order to figure things out it makes sense to mention locations.

[redeye86]

It seems that you are right, i tried to login from germany and from south korean via vpn. The login screens are different:

Login Germany:

Login South Korea:

[rafaelsampaio]

I access from Brazil and after https://accounts.google.com/ServiceLogin, I'm redirected to https://accounts.google.com/signin/v2/identifier for email and https://accounts.google.com/signin/v2/sl/pwd for password. All others URLs return 404.

[costastf]

Fun staff. Ok, so we need to try to combine a list of locations that do not work and I will try to figure out their method through a vpn and implement them in time.

[lufton]

I made it! Just replace your _authenticate method with following:

    def _authenticate(self):
        url = '{login_url}/ServiceLogin'.format(login_url=self._login_url)
        response = self._session.get(url)
        request_id = re.search(' data-initial-sign-in-data="%.*?"([\w-]{100,})"', response.text, re.S).group(1)

        self._session.headers.update({'Google-Accounts-XSRF': '1'})

        url = '{login_url}/_/signin/sl/lookup'.format(login_url=self._login_url)
        req = '["{email}","{request_id}"]'.format(email=self.email, request_id=request_id)
        response = self._session.post(url, data={'f.req': req})
        request_id = re.search('"([\w-]{100,})"', response.text, re.S).group(1)

        url = '{login_url}/_/signin/sl/challenge'.format(login_url=self._login_url)
        req = '["{request_id}",null,1,null,[1,null,null,null,["{password}",null,true]],[null,null,[2,1,null,1,"https://accounts.google.com/ServiceLogin",null,[],4,[],"GlifWebSignIn"],1,[null,null,[],null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,[]],null,null,null,true]]'.format(request_id=request_id, password=self.password)
        self._session.post(url, data={'f.req': req})

Also you can delete _initialize, _submit_email and _submit_password methods of Authenticator class. Please try that and verify if it is works for you.

[insajd]

I could not authenticate as well. Code from @lufton did help me get request on phone to confirm login from new device(import re must be added on the top as well), but cookie file that is generated does not work. I have old cookie file ~8KB, and new one (which I'm creating for another account) is generated ~1KB, and it doesn't work. @lufton do you get device locations working with this change?

[lufton]

@lufton do you get device locations working with this change?

Yes! It works for me. I can get response with location data. I create simple script because I don't know how to update source in Hass.io libraries:

#main.py

from locationsharinglib import Service
service = Service("example@gmail.com", "password", cookies_file=None)
for person in service.get_all_people():
    print(person)

And in console I got information about shared with me devices.

[insajd]

Thank you @lufton , I got it working for me at last with your code changes. Here are some notes:

1. This example(main.py) works for me if I disable 2FA. If I enable it back then it doesn't work for me. I also added cookies_file filename so that cookie is saved.
2. Although main.py works, moving created cookie file to HA didn't work. It threw error:

   
   Fri Oct 19 2018 16:13:06 GMT+0300 (Eastern European Summer Time)

Error setting up platform google_maps Traceback (most recent call last): File "/home/homeassistant/hass/lib/python3.5/site-packages/locationsharinglib/locationsharinglib.py", line 110, in _populate self._charging = data[13][0] TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/homeassistant/hass/lib/python3.5/site-packages/homeassistant/components/device_tracker/init.py", line 184, in async_setup_platform disc_info) File "/usr/lib/python3.5/asyncio/futures.py", line 380, in iter yield self # This tells Task to wait for completion. File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup future.result() File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result raise self._exception File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run result = self.fn(*self.args, **self.kwargs) File "/home/homeassistant/hass/lib/python3.5/site-packages/homeassistant/components/device_tracker/google_maps.py", line 47, in setup_scanner scanner = GoogleMapsScanner(hass, config, see) File "/home/homeassistant/hass/lib/python3.5/site-packages/homeassistant/components/device_tracker/google_maps.py", line 68, in init self._update_info() File "/home/homeassistant/hass/lib/python3.5/site-packages/homeassistant/components/device_tracker/google_maps.py", line 80, in _update_info for person in self.service.get_all_people(): File "/home/homeassistant/hass/lib/python3.5/site-packages/locationsharinglib/locationsharinglib.py", line 464, in get_all_people people = self.get_shared_people() + [self.get_authenticated_person()] File "/home/homeassistant/hass/lib/python3.5/site-packages/locationsharinglib/locationsharinglib.py", line 456, in get_authenticated_person None, File "/home/homeassistant/hass/lib/python3.5/site-packages/locationsharinglib/locationsharinglib.py", line 96, in init self._populate(data) File "/home/homeassistant/hass/lib/python3.5/site-packages/locationsharinglib/locationsharinglib.py", line 114, in _populate raise InvalidData locationsharinglib.locationsharinglibexceptions.InvalidData

So I changed `people = self.get_shared_people() + [self.get_authenticated_person()]` to `people = self.get_shared_people()` and now everything works for me 👍

[rafaalbelda]

The new _authenticate method from @lufton works for me (adding import re at the file begining)

I cleaned the locationsharinglib cache (just removed the pycache directory at /srv/homeassistant/lib/python3.5/site-packages/locationsharinglib)

I also removed the old .google_maps_location_sharing.cookies file and created a new one using:

from locationsharinglib import Service
service = Service("example@gmail.com", "password", cookies_file=.google_maps_location_sharing.cookies)

[ragenhe]

@rafaalbelda - can you show example of *adding import re at the file beginning ?

[lufton]

@rafaalbelda - can you show example of *adding import re at the file beginning ?

[lufton]

@costastf are you planing to update your lib. Do you need further help on fixing 2FA?

[costastf]

I need to check the changes and see if they break anything. I will work on this some time next week.

[StyxyDog]

Please try that and verify if it is works for you.

@lufton Thank you for your edits (https://github.com/costastf/locationsharinglib/issues/42?_pjax=%23js-repo-pjax-container#issuecomment-431195508 and import re. I applied these to my hassio install and was able to receive location information for the first time, in a terminal - not in hassio front end itself. I'm not familiar with hassio and have not been able to apply these changes as a permament fix. @costastf I'm in the UK using a standard personal gmail account (created specifically for this purpose). The account is not using 2FA. Hope that helps.

[KrzysztofHajdamowicz]

I can confirm that @lufton fix solves the problem and works for me.

[lufton]

Thank you for your edits (#42 and import re. I applied these to my hassio install and was able to receive location information for the first time, in a terminal - not in hassio front end itself. I'm not familiar with hassio and have not been able to apply these changes as a permament fix.

I can confirm that @lufton fix solves the problem and works for me.

Great! You're welcome. Glad to help somebody.

[costastf]

I have tested the code and it also works for me. There is no error handling and it completely breaks the CookieGetter and any 2fa banter. Any chance @lufton would like to actually address these issues according to the existing code style and make a proper PR that I can pull while also setting yourself as a contributor? If not, then this has to be parked for some time because I am very very busy to implement that myself at least for the upcoming two weeks, possibly more.

[costastf]

I am quite less than thrilled about the payload with all the hardcoded values.I would expect this to break really really quick with any change on their backend. It would be much better if you got the required values from the previous response so it is a bit more sturdy.

[lufton]

I am quite less than thrilled about the payload with all the hardcoded values.I would expect this to break really really quick with any change on their backend. It would be much better if you got the required values from the previous response so it is a bit more sturdy.

I realize that, I'll try to make it less changes dependent. I don't like regex search either, but I'm not a professional in Python and don't know if there is easier way to parse response values.

I have tested the code and it also works for me. There is no error handling and it completely breaks the CookieGetter and any 2fa banter. Any chance @lufton would like to actually address these issues according to the existing code style and make a proper PR that I can pull while also setting yourself as a contributor? If not, then this has to be parked for some time because I am very very busy to implement that myself at least for the upcoming two weeks, possibly more.

I'll try to do so ASAP.

[costastf]

Thanks and no rush! Most people can manually overwrite the method and get their things working. Lets make it nice, not fast. :)

[sagilo]

I made it! Just replace your _authenticate method with following:

    def _authenticate(self):
        url = '{login_url}/ServiceLogin'.format(login_url=self._login_url)
        response = self._session.get(url)
        request_id = re.search(' data-initial-sign-in-data="%.*?"([\w-]{100,})"', response.text, re.S).group(1)

        self._session.headers.update({'Google-Accounts-XSRF': '1'})

        url = '{login_url}/_/signin/sl/lookup'.format(login_url=self._login_url)
        req = '["{email}","{request_id}"]'.format(email=self.email, request_id=request_id)
        response = self._session.post(url, data={'f.req': req})
        request_id = re.search('"([\w-]{100,})"', response.text, re.S).group(1)

        url = '{login_url}/_/signin/sl/challenge'.format(login_url=self._login_url)
        req = '["{request_id}",null,1,null,[1,null,null,null,["{password}",null,true]],[null,null,[2,1,null,1,"https://accounts.google.com/ServiceLogin",null,[],4,[],"GlifWebSignIn"],1,[null,null,[],null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,[]],null,null,null,true]]'.format(request_id=request_id, password=self.password)
        self._session.post(url, data={'f.req': req})

Also you can delete _initialize, _submit_email and _submit_password methods of Authenticator class. Please try that and verify if it is works for you.

I can confirm this works for me as well (Israel)

[royduin]

It's working for me too 😎

[acarlo79]

I made it! Just replace your _authenticate method with following:

    def _authenticate(self):
        url = '{login_url}/ServiceLogin'.format(login_url=self._login_url)
        response = self._session.get(url)
        request_id = re.search(' data-initial-sign-in-data="%.*?"([\w-]{100,})"', response.text, re.S).group(1)

        self._session.headers.update({'Google-Accounts-XSRF': '1'})

        url = '{login_url}/_/signin/sl/lookup'.format(login_url=self._login_url)
        req = '["{email}","{request_id}"]'.format(email=self.email, request_id=request_id)
        response = self._session.post(url, data={'f.req': req})
        request_id = re.search('"([\w-]{100,})"', response.text, re.S).group(1)

        url = '{login_url}/_/signin/sl/challenge'.format(login_url=self._login_url)
        req = '["{request_id}",null,1,null,[1,null,null,null,["{password}",null,true]],[null,null,[2,1,null,1,"https://accounts.google.com/ServiceLogin",null,[],4,[],"GlifWebSignIn"],1,[null,null,[],null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,[]],null,null,null,true]]'.format(request_id=request_id, password=self.password)
        self._session.post(url, data={'f.req': req})

Also you can delete _initialize, _submit_email and _submit_password methods of Authenticator class. Please try that and verify if it is works for you.

I can confirm this works for me as well (Israel)

@sagilo have you been able to make your changes permanent in Hassio? Once you access the system in debug mode and replace the library, how do you save the docker?

thanks!

[sagilo]

I made it! Just replace your _authenticate method with following:

    def _authenticate(self):
        url = '{login_url}/ServiceLogin'.format(login_url=self._login_url)
        response = self._session.get(url)
        request_id = re.search(' data-initial-sign-in-data="%.*?"([\w-]{100,})"', response.text, re.S).group(1)

        self._session.headers.update({'Google-Accounts-XSRF': '1'})

        url = '{login_url}/_/signin/sl/lookup'.format(login_url=self._login_url)
        req = '["{email}","{request_id}"]'.format(email=self.email, request_id=request_id)
        response = self._session.post(url, data={'f.req': req})
        request_id = re.search('"([\w-]{100,})"', response.text, re.S).group(1)

        url = '{login_url}/_/signin/sl/challenge'.format(login_url=self._login_url)
        req = '["{request_id}",null,1,null,[1,null,null,null,["{password}",null,true]],[null,null,[2,1,null,1,"https://accounts.google.com/ServiceLogin",null,[],4,[],"GlifWebSignIn"],1,[null,null,[],null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,[]],null,null,null,true]]'.format(request_id=request_id, password=self.password)
        self._session.post(url, data={'f.req': req})

Also you can delete _initialize, _submit_email and _submit_password methods of Authenticator class. Please try that and verify if it is works for you.

I can confirm this works for me as well (Israel)

@sagilo have you been able to make your changes permanent in Hassio? Once you access the system in debug mode and replace the library, how do you save the docker?

thanks!

You need to edit the file where pip installs its libraries. In my case it's under /usr/local/lib/python3.6/site-packages/locationsharinglib/ You can also get a fixed copy of the locationsharinglib.py file and copy it to the relevant location in the docker container:

docker cp locationsharinglib.py <docker_container>:/usr/local/lib/python3.6/site-packages/locationsharinglib/locationsharinglib.py

[acarlo79]

thx, in my case after the docker restart, it seems to revert back to the original locationsharinglib.py

[lufton]

Okay, I created Pull Request, hopefully it looks good. I'm sorry @costastf it looks that I used completely different approach to get cookies that's why there were couple redundant methods and there are still methods that looks like they are hard-coded e.g:

data_key = data[0][10][0][0][23]['5004'][12]
data_tx_id = data[0][10][0][0][23]['5004'][1]

Sincerely I don't know what are all of those JSON Keys and Indexes stands for, but it works.

[sagilo]

thx, in my case after the docker restart, it seems to revert back to the original locationsharinglib.py

What command do you use for restarting the container?

[acarlo79]

tried: docker restart homeassistant

also tried from hassio: # hassio ha restart

should I do a docker commit?

[sagilo]

tried: docker restart homeassistant

also tried from hassio: # hassio ha restart

should I do a docker commit?

No, that should be fine.. Docker shouldn't restore the file unless the container is destroyed. Anyway, Looks like the fix is just around the corner... ;)

[omriasta]

I tried @lufton code but I am getting this error, anyone else see this?: Error setting up platform google_maps Traceback (most recent call last): File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/components/device_tracker/init.py", line 184, in async_setup_platform disc_info) File "/usr/local/lib/python3.6/asyncio/futures.py", line 331, in iter yield self # This tells Task to wait for completion. File "/usr/local/lib/python3.6/asyncio/tasks.py", line 244, in _wakeup future.result() File "/usr/local/lib/python3.6/asyncio/futures.py", line 244, in result raise self._exception File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 55, in run result = self.fn(*self.args, **self.kwargs) File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/components/device_tracker/google_maps.py", line 46, in setup_scanner scanner = GoogleMapsScanner(hass, config, see) File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/components/device_tracker/google_maps.py", line 66, in init self._update_info() File "/srv/homeassistant/homeassistant_venv/lib/python3.6/site-packages/homeassistant/components/device_tracker/google_maps.py", line 99, in _update_info ATTR_BATTERY_CHARGING: person.charging, AttributeError: 'Person' object has no attribute 'charging'

UPDATE: This was an issue on my end, for some reason locationsharinglib did not update and I was stuck on 3.0.3, had to upgrade pip before it went through. I have everything working now.

[ctdf]

Хорошо, я создал Pull Request , надеюсь, он выглядит хорошо. Извините @costastf. Похоже, что я использовал совершенно другой подход для получения файлов cookie, поэтому есть несколько избыточных методов, и есть еще методы, которые выглядят так, как будто они жестко закодированы, например:

data_key = data[0][10][0][0][23]['5004'][12]
data_tx_id = data[0][10][0][0][23]['5004'][1]

С уважением, я не знаю, что это за те ключевые слова и индексы JSON, но это работает.

You forgot to import re and replace def _authenticate(self):...

Thank you, after making all your corrections, it worked for me. But I had to disable 2FA.

[lufton]

Хорошо, я создал Pull Request , надеюсь, он выглядит хорошо. Извините @costastf. Похоже, что я использовал совершенно другой подход для получения файлов cookie, поэтому есть несколько избыточных методов, и есть еще методы, которые выглядят так, как будто они жестко закодированы, например:

data_key = data[0][10][0][0][23]['5004'][12]
data_tx_id = data[0][10][0][0][23]['5004'][1]

С уважением, я не знаю, что это за те ключевые слова и индексы JSON, но это работает.

You forgot to import re and replace def _authenticate(self):...

Thank you, after making all your corrections, it worked for me. But I had to disable 2FA.

Actually last Pull Request I made has 2FA feature working and does not require regex lib (import re).

[jdecker91]

I got this working without 2FA and I just wanted to share what I did in case anyone else is still having issues. I updated my Home Assistant to 0.81.1 today and this was still failing. the google_maps sharing has never worked for me. Here's what I did.

1. Changed the people variable in the get_all_people method as @insajd did. (Not sure if this is needed as I still saw errors after this) https://github.com/costastf/locationsharinglib/issues/42#issuecomment-431362416

2. Replaced the _authenticate method with the code that @lufton provided. (This gave me an error along the lines of saying Regex is not found.) https://github.com/costastf/locationsharinglib/issues/42#issuecomment-431195508

3. I imported the Regex module in the locationsharinglib.py by adding import re to the the script

4. Then I deleted the pycache directory in the locationsharinglib directory (/srv/homeassistant/lib/python3.5/site-packages/locationsharinglib) and deleted the .google_maps_location_sharing.cookins file from the main home assistant directory https://github.com/costastf/locationsharinglib/issues/42#issuecomment-431413384

5. Restarted Home Assistant and my device tracker started showing up.