home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
71.14k stars 29.81k forks source link

TUYA HUB is trying to connect to DEAD "apigw.tuyaeu.com" address, so TUYA integration is not working at all. #123375

Closed AnunnakiNibiru closed 1 week ago

AnunnakiNibiru commented 1 month ago

The problem

Few weeks ago, I installed new "Home Assistant Green" device, and tried to add TUYA integration. It was not working and I was not having time to solve that then, so I was not using HA till few days ago. So few days ago I tried it again, but with the same negative result as before. I even performed "factory recovery" via SD card to start "fresh".

I think, TUYA is using non-functional API FQDN address (at least for central European region): apigw.tuyaeu.com instead (I think correct one): openapi.tuyaeu.com

What I did/tested so far: After factory reset I started fresh. I immediately as first step tried add TUYA integration (the one already available via GUI). I added user code (from mobile app), via mobile app QR scan I confirmed "Home Assistant" user. In my "Tuya app" on mobile phone > settings > account and security > and under "Third-Party Authorization Management" I have now "Home Assistant" with 2 permissions: "Access your account information" "Access the control of your device" So, I assume, "authorization" steps are done correctly.

But ... inside Home Assistant, integration never finished correctly, even after few minutes of waiting it show "succesfully" but window does not contain any other information like it should (according other Youtube videos I have seen). In Home Assistant under Tuya integration I see: Needs attention No devices or entities Failed to set up: Check the logs ...and... Hubs No entires

I tried "Enable debug logging" and it seems to me that it tries to connect to: File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 594, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='apigw.tuyaeu.com', port=443): Max retries exceeded with url: /v1.0/m/life/users/homes (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0xffff65a5b080>, 'Connection to apigw.tuyaeu.com timed out. (connect timeout=None)'))

I investigated, that for my location (Czech Republic) I am at "Central Europe Data Center". That FQDN apigw.tuyaeu.com is translated to IPs: 3.123.196.97 and 18.193.166.37 . I even tried "tcpdump" communication from HA, my router (home&work), from laptop (my work), and even from my mobile phone (5G) ... (note that all 3 devices are using different ISP operator, and communicating out via different public addresses, so it is no connection issue on my side), but in every case: That packet is send (with SYN tcp flag) to one of those two IP addresses (under FQDN apigw.tuyaeu.com), but reply is never received. There should be at least TCP SA or R packet received. But no response at all.

In some documentation on TUYA is mentioned, that "Central Europe Data Center" should use different API FQDN: openapi.tuyaeu.com (that one is responding to wget and ping correctly). (see here documentation with all API FQDNs for all regions: https://developer.tuya.com/en/docs/iot/api-request?id=Ka4a8uuo1j4t4 )

So my question is ... are you 100% sure, that you are using correct API address? And if so, why that address is not responding (probably for all users in Czech Republic and maybe for all in Central European region, where mentioned DEAD FQDN address is used)?

Thank you.

I teste it in 2024.7.4 and 2024.8.0

Additional information

If you wish, I can share my desktop with access to home assistant green and even share my phone app, if you wish investigate/test something further. If you need more info, just let me know.

What version of Home Assistant Core has the issue?

core-2024.8.0

What was the last working version of Home Assistant Core?

Never worked in my region.

What type of installation are you running?

Home Assistant OS

Integration causing the issue

TUYA as whole integration

Link to integration documentation on our website

No response

Diagnostics information

File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 594, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='apigw.tuyaeu.com', port=443): Max retries exceeded with url: /v1.0/m/life/users/homes (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0xffff65a5b080>, 'Connection to apigw.tuyaeu.com timed out. (connect timeout=None)'))

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 1 month ago

Hey there @tuya, @zlinoliver, @frenck, mind taking a look at this issue as it has been labeled with an integration (tuya) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `tuya` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign tuya` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)


tuya documentation tuya source (message by IssueLinks)

tomgat6 commented 1 month ago

I am using Home Assistant Tuya integration for some time now. The address apigw.tuyaeu.com is correct and working for me (located in Austria). I can monitor traffic to the IPs you mentioned, e.g., 18.193.166.37, to and from my HA system through my firewall, so the apigw is generally active and responding. I have about 40 devices using this integration and it is very stable.

Note that the openapi.tuyaeu.com address is from another variant of the tuya api (the "openapi"), which is a paid service for OEMs. Contrary the apigw.tuyaeu.com address is for a tuya api with reduced functionality especially for Home Assistant and free to use.

AnunnakiNibiru commented 4 weeks ago

Thanks for reply. But still, at least from CzechRepublic (3 ISPs, 1 global, 2 around Prague) is that IP address dead, not-responding. Perhaps for Czech Republic, another API address is needed? Or where/how can I make official Tuya working in HA? That was whole reason for buying HA :(. Last address to respond is from "Amazon" it seems.

Tracing route to apigw.tuyaeu.com [18.193.166.37] over a maximum of 30 hops:

... 3 1 ms 1 ms 1 ms 10.102.223.161 4 2 ms 81.201.48.7 5 2 ms 1 ms 1 ms klfree-vl3804.nfx.cz [78.108.106.6] 6 2 ms 2 ms 2 ms asr9904-vl3999-nfx2.nfx.cz [81.201.48.158] 7 2 ms 2 ms 2 ms nix4.amazon.com [91.210.16.120] 8 2 ms 2 ms 2 ms 52.93.43.122 9 2 ms 3 ms 2 ms 52.93.43.125 10 Request timed out. 11 Request timed out. 12 Request timed out. 13 Request timed out. 14 Request timed out. 15 Request timed out. 16 Request timed out. 17 Request timed out. 18 Request timed out. 19 Request timed out. 20 Request timed out. 21 Request timed out. 22 Request timed out. 23 Request timed out. 24 Request timed out. 25 Request timed out. 26 Request timed out. 27 Request timed out. 28 Request timed out. 29 Request timed out. 30 * Request timed out.

tomgat6 commented 4 weeks ago

Thanks for the answer! As you correctly mentioned, the api server for 18.193.166.37 appears to be in the Amazon Web Services cloud (AWS). These systems respond only to whitelisted IP ranges. It may happen, that your ISPs or even country is not whitelisted, although the global tuya login system directed you to that regional service for the EU. To make a final check: If you open a browser and type the address "https://18.193.166.37" you get no response? If I do the same, I get an error message (because that request of course is no valid API call) like this: '{"message":"no Route matched with those values"}'

My guess would be that this is a misconfiguration of this AWS service.

The Tuya integration is maintained here: https://github.com/tuya/tuya-device-sharing-sdk Maybe you can get in touch with the maintainers, and they know who operates the AWS service for apigw.tuyaeu.com and get their position on your issue.

Just my 2ct.

AnunnakiNibiru commented 4 weeks ago

Thank you.

I created ticket: https://github.com/tuya/tuya-device-sharing-sdk/issues/18 I already contacted "Neutral czFree eXchange" which is used by many Czech ISP providers as peering center, and they confirmed the same problem.

Both IP addresses (translated from mentioned FQDN) are not sending back any packet, so even not responding to TCP SYN with TCP SA, that means TCP connection is not established at all. Someone at "amazon" is dropping request (SYN packet) or responses (any packet) from those two addresses. Probably all users and ISP providers from Czech Republic, which are connected via peering centrum "Neutral czFree eXchange" (https://nfx.cz), have probably the same problem.

I am awaiting for answer from "https://github.com/tuya/tuya-device-sharing-sdk/issues/18" , hope they provide any info, whom to contact further.

davidmartinek-dev commented 2 weeks ago

@AnunnakiNibiru Did you get a reply, or found an another solution? I have the same ISP as you do

AnunnakiNibiru commented 2 weeks ago

From what I understand so far, the problem is on TUYA side somewhere in AMAZON EU services, where they (TUYA) have filter for that "apifw.tuyaeu.com" IP's, to drop communication's from other than Central European countries. And they are missing some subnets/IP ranges from many Czech Republic ISP's, and perhaps also from other countries (Germany, with 212.79.0.0 at least).

No, no one has contacted me, and I don't have any information about who is responsible for fixing that problem.

All my devices are TUYA (some WIFI, some ZigBee). For me, Home Assistant is not working with Tuya at all, so is lying on the shelf. Wasted money.

AnunnakiNibiru commented 2 weeks ago

@davidmartinek-dev

FYI: I contacted some TUYA support, and they pointed me to place, where I created new ticket, which I did. Now, I am waiting ... hope someone will solve this there: https://service.console.tuya.com/8/3/detail?id=T202408270001&source=support_center

AnunnakiNibiru commented 2 weeks ago

Anyone, who thinks that TUYA API "apigw.tuyaeu.com" is "dead" and not responding at all, should try to investigate it more. Then you should create a ticket here: https://service.console.tuya.com/8/3/list?source=support_center (you need to register there, if you don't have account already) "New general ticket" >>> "Cloud Development" And in parallel, open a ticket with your internet connection provider (ISP), on the same issue.

Below are some useful commands (for Windows 11 users) in PowerShell, which you can run, and COPY and PASTE outputs from commands below to a TXT file, and upload that TXT file to the ticket. It can help them to investigate and hopefully even solve the problem.

If you already have "PowerShell 7.4.5" and "nMap 7.80" apps installed, go straight to the test commands.

This command will start the update of the "winget" installer and then all other applications. You can run it from "CMD" or "PowerShell" or maybe even other shells, which you have on your Windows: winget update --verbose --force --include-unknown --all

If you don't have "powershell" 7.4.5 installed, install it with the command: winget install --id "Microsoft.PowerShell"

If you don't have "nmap" 7.80 installed (we need nping tool from it), install it with the command winget install --id "Insecure.Nmap"

After installation completed, than you need to close "Command line" or "PowerShell" or "Terminal", and start "PowerShell" again. Start PowerShell 7.4.5

Test commands: suitable for testing the functionality of the connection to the TUYA API:

This command will attempt to connect to the API using an HTTP|HTTPS query: Invoke-WebRequest -Uri "https://apigw.tuyaeu.com" -UseBasicParsing -TimeoutSec 29 -Verbose

This command attempts to establish a TCP connection to the API: Test-NetConnection -ComputerName "apigw.tuyaeu.com" -Port 443 -InformationLevel "Detailed"

This command tries to use TTL decrement to determine which IP address in the path to the API will respond last: Test-NetConnection -ComputerName "apigw.tuyaeu.com" -TraceRoute -Hops 30 -InformationLevel "Detailed"

This command does the same thing as above, just with TCP on port 443: nping --tcp --dest-port 443 --traceroute "apigw.tuyaeu.com" -v2

Alternatively, you can share the output of at least "nping" command, and your public IP here, and I will attach it to my ticket to fix it.

AnunnakiNibiru commented 2 weeks ago

TUYA support closed my ticket. According to them, they don't see incoming traffic from my public IP address in the logs when trying to access "apigw.tuyaeu.com". They only see my communication with "openapi.tuyaeu.com", but it works normally.

When I switch to ISP Vodafone CR, the connection to "apigw.tuyaeu.com" works normally.

I don't know how much I can trust TUYA support, what did they actually check and where (they wanted me to test things that are clear to a technically knowledgeable person that they can't work, but maybe it was a mandatory automatic exercise, they also can't know what kind of "idiot" is bothering them, which i understand ). In any case, they confirmed to me twice that they checked it with them at the input in the log records.

At the moment I'm back in the starting position. I opened a new ticket with my ISP and asked him to check from his various public subnets whether it would work for him at least from some addresses or not, and then let him solve it further with his ISP/peering center. I hope that we will gradually figure out who is responsible for the malfunction.

It would be good for HomeAssistant, instead of writing a nonsensical message/error, to clearly state that the "apigw.tuyaeu.com" API is not communicating at all, and for the user to check the network connection and contact their ISP.

I will inform you about the further development of the situation.

tomgat6 commented 2 weeks ago

I'm not sure what "logs" the Tuya support is looking at, but when they are looking at their API (gateway) server logs, it's not surprising that they see no incoming traffic from your IP. The apigw.tuyaeu.com is an EC2 (elastic compute cloud) instance in AWS (Amazon web services):

>ping apigw.tuyaeu.com

Ping wird ausgeführt für apigw.tuyaeu.com [18.193.166.37] mit 32 Bytes Daten:
...

> ping -a 18.193.166.37

Ping wird ausgeführt für ec2-18-193-166-37.eu-central-1.compute.amazonaws.com [18.193.166.37] mit 32 Bytes Daten:

If the EC2 instance's network interface / load balancer / etc. is blocking a given IP (range), the underlying server or service is not seeing the trafic at all - as it seems in your case. You can point Tuya support to this possibility, and, e.g., to documentation like this: https://repost.aws/knowledge-center/ec2-block-or-allow-ips, maybe they overlooked this in their investigations.

AnunnakiNibiru commented 2 weeks ago

I asked them exactly the same thing. And they confirmed to me that they don't block anything anywhere. I also wanted to know what exactly they verified it on, just to be sure, but they couldn't tell me even that, not a single technical detail that I could rely on, so I had minor "doubts" about the detail and conclusion of their investigation. But I believe that they are experts in their field, and for now I have no choice but to trust them. Nevertheless, I opened the ticket again and asked where they were checking it and what if they provided some technical details, and I informed them that I was waiting (for the second time) to check the status with my ISP (and above).

The only alternative for me is to change public IP, change internet provider, or for IoT stuff pay for a dedicated public VPN/anonymizer (through which it works, but costs extra money every month).

beolink commented 1 week ago

Hi, I have the exact same problem in Sweden. If I use VPN or a 4G adapter it works perfectly but with my normal ISP I get the same issue: PublicIP 217.31.190.xxx

AnunnakiNibiru commented 1 week ago

A second check through my ISP revealed a problem on some FW down the road where some CVE-related protection was enabled ( "https://access.redhat.com/security/vulnerabilities/tcpsack" ). After a modification on their end it works now. They also confirmed to me that it affected several public subnets. The problem is now solved. I am closing this ticket. Have a nice day.