search

Chouffy / home_assistant_tgtg

TooGoodToGo items stock as sensor in Home Assistant
MIT License
65 stars 18 forks source link

Get tokens fails because of a captcha #80

Closed euro2 closed 12 months ago

euro2 commented 1 year ago

System Health details

-

Checklist

Describe the issue

-

Reproduction steps

run the get tokens py

Debug logs

python3 tgtg_get_tokens.py 
Type your email linked to your TGTG account: xxx@hotmail.com
Using version 23.4.12
Traceback (most recent call last):
  File "/root/tgtg_get_tokens.py", line 13, in <module>
    tgtgClient.get_credentials()
  File "/usr/local/lib/python3.10/dist-packages/tgtg/__init__.py", line 90, in get_credentials
    self.login()
  File "/usr/local/lib/python3.10/dist-packages/tgtg/__init__.py", line 181, in login
    raise TgtgLoginError(response.status_code, response.content)
tgtg.exceptions.TgtgLoginError: (403, b'{"url":"https://geo.captcha-delivery.com/captcha/?initialCid=AHrlqAAAAAMAdRgF1adAg_gA2WmLtw==&cid=86PDjw6-EpSxHPjvbU43ToKotX~i8HHzU1h2qr4e-FnmR_iSetqOR-6lSkG2eLprcf~cI8nfz7tzYKidv-OTGGfKhYlmg7yQaRlfYwVm8bL-r56HapF5Z-c6uGd1336&referer=https%3A%2F%2Fapptoogoodtogo.com%2Fapi%2Fauth%2Fv3%2FauthByEmail&hash=1D42C2CA6131C526E09F294FE96F94&t=fe&s=35587&e=6b1d5728fb084e8d880c6e0c40adab58986febb54699b2c4d83229fa46fd6df9"}')

Diagnostics dump

No response

euro2 commented 1 year ago

I now tried a fresh docker run to get tokens. Same result. It errors on the captch url. If you paste it in the browser you get an actual captcha.... Maybe too many polls to their server?

joshuaferrara commented 1 year ago

maybe a dumb question, but are you running v5.2.0 of home_assistant_tgtg? i requested new tokens using the underlying Python library @ its latest version (see their README.md), upgraded from v5.1.0 to v5.2.0 in HACS w/o restarting, updated my HA configuration to use the new tokens, and then restarted everything. after that I'm back in action.

tjorim commented 1 year ago

I now tried a fresh docker run to get tokens. Same result. It errors on the captch url. If you paste it in the browser you get an actual captcha.... Maybe too many polls to their server?

Define 'fresh docker run', did you also do an clean build (as it might have cached some layers. Try this: docker build --no-cache https://github.com/Chouffy/home_assistant_tgtg.git#main --tag "homeassistant_tgtg_tokens:latest".

euro2 commented 1 year ago

I now tried a fresh docker run to get tokens. Same result. It errors on the captch url. If you paste it in the browser you get an actual captcha.... Maybe too many polls to their server?

Define 'fresh docker run', did you also do an clean build (as it might have cached some layers. Try this: docker build --no-cache https://github.com/Chouffy/home_assistant_tgtg.git#main --tag "homeassistant_tgtg_tokens:latest".

A fresh run is quite vaque isnt it :)...

What i ment is that i installed docker in a new LXC container and issued the commands as described in installation 1b: docker build https://github.com/Chouffy/home_assistant_tgtg.git#main --tag "homeassistant_tgtg_tokens:latest" docker run --rm -it homeassistant_tgtg_tokens

EDIT: it works now. The captcha thought i was a bot yesterday..... I now have edited my old WORKING config to match the newly obtained one but my sensors are all still unavailable. I have updated the HACS to 5.2.0 a couple of days ago as well.... (+ full reboot)

{ "domain": "tgtg", "name": "TooGoodToGo", "codeowners": ["@chouffy", "@tjorim"], "documentation": "https://github.com/Chouffy/home_assistant_tgtg", "iot_class": "cloud_polling", "issue_tracker": "https://github.com/Chouffy/home_assistant_tgtg/issues", "requirements": ["tgtg==0.16.0"], "version": "5.2.0" }

Home Assistant 2023.5.4 Supervisor 2023.04.1 Operating System 10.2 Frontend-versie: 20230503.3 - latest

So still troubleshooting the unavailable sensors.

euro2 commented 1 year ago

New info: the Home assistant log says captcha again:

`Logger: homeassistant.components.sensor Source: custom_components/tgtg/sensor.py:189 Integration: Sensor (documentation, issues) First occurred: 10:04:05 (1 occurrences) Last logged: 10:04:05

Error while setting up tgtg platform for sensor Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 304, in _async_setup_platform await asyncio.shield(task) File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/config/custom_components/tgtg/sensor.py", line 76, in setup_platform add_entities([TGTGSensor(each_item_id)]) File "/config/custom_components/tgtg/sensor.py", line 96, in init self.update() File "/config/custom_components/tgtg/sensor.py", line 189, in update self.tgtg_answer = tgtg_client.get_item(item_id=self.item_id) File "/usr/local/lib/python3.10/site-packages/tgtg/init.py", line 276, in get_item self.login() File "/usr/local/lib/python3.10/site-packages/tgtg/init.py", line 152, in login self._refresh_token() File "/usr/local/lib/python3.10/site-packages/tgtg/init.py", line 138, in _refresh_token raise TgtgAPIError(response.status_code, response.content) tgtg.exceptions.TgtgAPIError: (403, b'{"url":"https://geo.captcha-delivery.com/captcha/?initialCid=AHrlqAAAAAMAjokilHqJT8cA2WmLtw==&cid=73DGL~8P1WGJMtY3~PkNluG_n8aH7T3hn8JzlPC~DPsg6SGlH8~E4_4Ec9fhe56p7eD6ZZ9T3he0GuJXiMG80dv1_xVtulrutO7riDwH6_5NbE-4dnofL3jIdSs0e8Nr&referer=https%3A%2F%2Fapptoogoodtogo.com%2Fapi%2Fauth%2Fv3%2Ftoken%2Frefresh&hash=1D42C2CA6131C526E09F294FE96F94&t=fe&s=35558&e=1f8a41279d5b89aed8e94e08de3b9b0ba52cb7b17ec8fd7e1699e673dbb10f51"}') `

So TGTG thinks im spamming? My refresh is 300sec.

Once i started my android app on the same IP as HA, the app gave me the captcha as well. So that should be it. They changed something to stops excessive polling.

euro2 commented 1 year ago

nope. keep getting captcha's again with docker run --rm -it homeassistant_tgtg_tokens And the one time this morning it worked and i got the tokens, i pasted over the tokens to HA, but then i i had captchas in the Homeassistant tgtg error logs. So i think im flagged as spammer or something.

On android i cleared cache and storage of the app and started over. Again a captcha before i could continue with email. After that the app works fine.

I not sure what to do at the moment. Ill wait a few days before trying again. In HA i remove the HACS addon so it wont poll the tgtg api

Alfagek commented 1 year ago

Here now also problems with version 5.2 on HA 2023.5.4.

Did have this error first in HA log

`Logger: homeassistant.helpers.entity Source: custom_components/tgtg/sensor.py:189 Integration: tgtg (documentation, issues) First occurred: 13:11:45 (200 occurrences) Last logged: 16:06:16

Update for sensor.tgtg_xxxxxxxxxxxxxxxxxxxxxx fails Update for sensor.tgtg_xxxxxxxxxxxxxxxxxxxxxx fails Update for sensor.tgtg_xxxxxxxxxxxxxxxxxxxxxx fails Update for sensor.tgtg_xxxxxxxxxxxxxxxxxxxxxx fails Update for sensor.tgtg_xxxxxxxxxxxxxxxxxxxxxx fails Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 554, in async_update_ha_state await self.async_device_update() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 781, in async_device_update await hass.async_add_executor_job(self.update) File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/config/custom_components/tgtg/sensor.py", line 189, in update self.tgtg_answer = tgtg_client.get_item(item_id=self.item_id) File "/usr/local/lib/python3.10/site-packages/tgtg/init.py", line 287, in get_item raise TgtgAPIError(response.status_code, response.content) tgtg.exceptions.TgtgAPIError: (403, b'{"url":"https://geo.captcha-delivery.com/captcha/?initialCid=AHrlqAAAAAMAtC-4u-6q-ZsAVPUifw==&cid=5tBB_ZqEGa4qKLQdSZY-aErGCQ7kzOCaftnig7A-WITA9DLKzuIQu_cbODD-B9nirRHYmOFZvVYwT1WLapEk7-Nr1c15d94mhcB5hovNNIMN0t7aPUU8dGPRXFmdJUzB&referer=https%3A%2F%2Fapptoogoodtogo.com%2Fapi%2Fitem%2Fv8%2F53574&hash=1D42C2CA6131C526E09F294FE96F94&t=fe&s=35558&e=f73668f01a34c24a90c7911b0116da56e864aca6dbbd9b332046b85022e4601a"}')`

Then i did try to get new tokens in terminal. Can start the "get tokens", but after entering my mail adres ill get the next error: ➜ tgtg python3 tgtg_get_tokens.py Type your email linked to your TGTG account: <MY MAIL ADRES> Using version 23.5.1 Traceback (most recent call last): File "/config/tgtg/tgtg_get_tokens.py", line 11, in <module> tgtgClient.get_credentials() File "/usr/lib/python3.11/site-packages/tgtg/__init__.py", line 90, in get_credentials self.login() File "/usr/lib/python3.11/site-packages/tgtg/__init__.py", line 181, in login raise TgtgLoginError(response.status_code, response.content) tgtg.exceptions.TgtgLoginError: (403, b'{"url":"https://geo.captcha-delivery.com/captcha/?initialCid=AHrlqAAAAAMAngbzfJ75Gb8AVPUifw==&cid=4Wh6TDs5~PIfjvdQ527wcGACGk~DqRsvsxbFIm-Nb9SZlK6etsWNOVvkVC7yZjdPVfFAvPBlS2sKtlawh8XS7_9xgv4zphorYoP2AVpNVPmu_GdF~aQLbpiepEeZTwNd&referer=https%3A%2F%2Fapptoogoodtogo.com%2Fapi%2Fauth%2Fv3%2FauthByEmail&hash=1D42C2CA6131C526E09F294FE96F94&t=fe&s=35587&e=726300966a9904a2190c56879aacd14e94f93e4e461247581dc219889f763395"}')

Also before i did do the new "get token" command, when i did opened the app on my iPhone, i did get a "captcha" puzzle. Never had that before.

euro2 commented 1 year ago

I hope this hurdle doesnt kill the Integration :(

Alfagek commented 1 year ago

It’s working here again. I think it was some kind of Ip block. Only strange thing that the app did work on the same WiFi network.

Did do this afternoon a new “get token” past it in to de yaml. Restart HA and error was gone. And sensors are working again (with boxes)

Edit:

It did work for a few hours. And now I do get errors again. And when i start the app, I did get again the captcha puzzle. Did have a pull for 60 sec. So adjust it to 120sec and not restarted HA. Let’s see what will happen.

euro2 commented 1 year ago

please keep us updated on your progress

Alfagek commented 1 year ago

Still no succes. 2 days ago i did get a message from tgtg addon on my phone. And after that it was down again. And again the captcha puzzle in the tgtg app. So didn't do anything with the intergration in ha. It wasn't doing anything (i think) because i did get the error, intergration is not working because of the captcha 403 error.

Today i started the app, all working. So restart HA and error captcha 403. Opening the app on phone capctha puzzle again :-(. So did that, restart HA en and again same error 403.

Today i disabled tgtg in HA. And will wait 24 hours, before i turn it on again. See if that wil do something.

A other strange thing i did mention in HA with TGTG. If i go to terminal and i give the command to get tokens. I do get the next error: ➜ tgtg python3 tgtg_get_tokens.py Traceback (most recent call last): File "/config/tgtg/tgtg_get_tokens.py", line 4, in <module> from tgtg import TgtgClient ModuleNotFoundError: No module named 'tgtg'

When i "reinstall tgtg" in terminal, its working again. I also did had this a few days ago. I can't remeber if this was also before i did have alle the "tgtg errors". Working now with 2023.6.1

Alfagek commented 1 year ago

Well i don't have a clue how to handle this. I'll hope that @tjorim has a few suggestions to try.

Before i did start this afternoon, i started the app on my phone: No captcha puzzle or what so ever.

I did try to get tokens by using "get tokens". After entering mail adres: 403 error. Then i did see on my phone, a captcha puzzle was there. So did puzzle, closed app en did try again with same result. Then i removed tgtg install with "pip uninstall tgtg". And used "pip install tgtg", and tgtg 16 was installing again. Then i did try again to get the token, same 403 error, and no captcha on the app. I did try to use the "docker" way on a other virtual machine with HA (on Advanced ssh & Web Terminal) (my test system), same result.

So i'm totally lost. All did on the same network.

euro2 commented 1 year ago

i think TGTG noticed the 'api-spam' and introduced captchas.... :(

Alfagek commented 1 year ago

Update:

I disabled last weekend the config in HA for Tgtg. And only did use the Tgtg app twice this week. And no captcha puzzle.

So today i did a new attempt: used “get token” and after entering mail adres, no 403 error 😱

then copie all the thing in to my yaml file. And adjust the pull time to 300 seconds. Made the yaml file active again. And restarted HA, no errors 👌🏻. So that did look good.

After about 15 minutes the first packs did come visible in HA. And that was about 16:00 my time. It’s now 21:00 and it’s still running without a 403 error.

So it’s looks like it’s going the right way! fingers crossed

Alfagek commented 1 year ago

It was working great with a 5 minute pull.

But changed the pull this afternoon to 1 minute and I do get the 403 back again.

So I’m disable the sensor again and will wait 2 days en place it back with a pull of 5 minutes. Better 5 minutes then a not working addon ;-)

Strauman commented 1 year ago

It was working great with a 5 minute pull.

But changed the pull this afternoon to 1 minute and I do get the 403 back again.

So I’m disable the sensor again and will wait 2 days en place it back with a pull of 5 minutes. Better 5 minutes then a not working addon ;-)

Did it work?

Alfagek commented 1 year ago

It was working great with a 5 minute pull. But changed the pull this afternoon to 1 minute and I do get the 403 back again. So I’m disable the sensor again and will wait 2 days en place it back with a pull of 5 minutes. Better 5 minutes then a not working addon ;-)

Did it work?

Sorry for the late reaction (holiday). But it did work. It’s running now for a long period and have it set to 180 seconds.

Strauman commented 1 year ago

Worked for me too. Anything under 300 sec triggers captcha for me.

Strauman commented 1 year ago

A other strange thing i did mention in HA with TGTG. If i go to terminal and i give the command to get tokens. I do get the next error: ➜ tgtg python3 tgtg_get_tokens.py Traceback (most recent call last): File "/config/tgtg/tgtg_get_tokens.py", line 4, in <module> from tgtg import TgtgClient ModuleNotFoundError: No module named 'tgtg'

When i "reinstall tgtg" in terminal, its working again. I also did had this a few days ago. I can't remeber if this was also before i did have alle the "tgtg errors". Working now with 2023.6.1

I suspect that could because the folder name Is the same name as the module. So if you rename /config/tgtg/ to /config/toogood/ it might fix it :)

lacrima08 commented 12 months ago

Hi,

Got errors too, ask me for captcha

Last logged: 18:23:56

Update for sensor.tgtg_xxxxxxx fails Update for sensor.tgtg_xxxxxxx fails Update for sensor.tgtg_xxxxxxx fails Update for sensor.tgtg_xxxxxxx fails Update for sensor.tgtg_xxxxxxx fails Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 699, in async_update_ha_state await self.async_device_update() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 940, in async_device_update await hass.async_add_executor_job(self.update) File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/tgtg/sensor.py", line 189, in update self.tgtg_answer = tgtg_client.get_item(item_id=self.item_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/tgtg/init.py", line 287, in get_item raise TgtgAPIError(response.status_code, response.content) tgtg.exceptions.TgtgAPIError: (403, b'{"url":"https://geo.captcha-delivery.com/captcha/?initialCid=AHrlqAAAAAMAA8S4ze7nR8MAUkJApQ==&cid=3WSwSWOHo1arVZ4K2MMPeEx30T6XGSFfkhn2OIRLuSiDZxN3uuT_XjxxrdPlQ4-991rvdBfYujeW2le8ACflyKfK7V5PlSMz0OSsClBGlmWCYiRcCwlI1hpF1EnBzCWg&referer=https%3A%2F%2Fapptoogoodtogo.com%2Fapi%2Fitem%2Fv8%2F50108&hash=1D42C2CA6131C526E09F294FE96F94&t=fe&s=35558&e=339dd2286a20985f6ff651b1be55fe65ce034c76b4b385c98360aaadfec8be14"}')

And the python script on windows got the same :

tgtg_get_favorites_item_id.py Type your email linked to your TGTG account: email@example.com Using version 23.9.2 Traceback (most recent call last): File "C:\Users\USER\Downloads\tgtg_get_favorites_item_id.py", line 6, in from tgtg_get_tokens import tgtgClient File "C:\Users\USER\Downloads\tgtg_get_tokens.py", line 13, in tgtgClient.get_credentials() File "C:\Users\USER\AppData\Local\Programs\Python\Python312-32\Lib\site-packages\tgtg__init__.py", line 87, in get_credentials self.login() File "C:\Users\USER\AppData\Local\Programs\Python\Python312-32\Lib\site-packages\tgtg__init__.py", line 178, in login raise TgtgLoginError(response.statuscode, response.content) tgtg.exceptions.TgtgLoginError: (403, b'{"url":"https://geo.captcha-delivery.com/captcha/?initialCid=AHrlqAAAAAMAB280dAYzuyIAUkJApQ==&cid=5HUfPzsUXlqS4paE4bcK53HyQjRTZOQCIhIul7vZMcbdfSwjsGUVW713pgMkFoi921cMSYG21I9~C~br7R4slRAOl-29yj3lXm3OB13mekRy6idDoQ25QorVoXVcA3k&referer=https%3A%2F%2Fapptoogoodtogo.com%2Fapi%2Fauth%2Fv3%2FauthByEmail&hash=1D42C2CA6131C526E09F294FE96F94&t=fe&s=35587&e=c6746ea4e22ebaa54df9da5b07063ef584a71c03b1dd9de6101c3b1cebdf3b0f"}')

tjorim commented 12 months ago

We can't do anything about the captcha but it's probably related to polling too often (scan_interval). The default we currently recommend is 900 seconds. I'm also not sure if it is limited by the total calls per day or too many in a short period.

Some users reported similar issues in the upstream library we use: https://github.com/ahivert/tgtg-python/issues/250. I pushed the change here https://github.com/Chouffy/home_assistant_tgtg/pull/84 and published version 5.3.0: https://github.com/Chouffy/home_assistant_tgtg/releases/tag/5.3.0. I'll close this issue for now as we don't have any control over it.