search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
72.07k stars 30.18k forks source link

Ecowitt Integration - HTTPS requirements non functional #93000

Open JohnMcLear opened 1 year ago

JohnMcLear commented 1 year ago

The problem

Duplicate of https://github.com/garbled1/homeassistant_ecowitt/issues/154

I'm running HA Blue hardware w/ HASS and want to use HTTPS to make Ecowitt plugin (core) work.

Since the plugin went into core HA the TLS/SSL was dropped and the requirement is to use NGINX to reverse proxy HTTPS > HTTP.

Before I had port 4199 > 4199 forwarded(at the router) to the HA Blue box and it was working fine (using custom plugin). I modified the new path to match the path specified by the plugin at time of installation/configuration and I have no data coming through and no errors in the logs.

Given that I expose https for core without a plugin what are my options here?

Related to https://github.com/garbled1/homeassistant_ecowitt/issues/149 but I wanted to create a new issue to remove the noise

What version of Home Assistant Core has the issue?

core-2023.5.2

What was the last working version of Home Assistant Core?

n/a

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Ecowitt

Link to integration documentation on our website

https://www.home-assistant.io/integrations/ecowitt/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 1 year ago

Hey there @pvizeli, mind taking a look at this issue as it has been labeled with an integration (ecowitt) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `ecowitt` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign ecowitt` Removes the current integration label and assignees on the issue, add the integration domain after the command.

(message by CodeOwnersMention)

ecowitt documentation ecowitt source (message by IssueLinks)

veilofsecurity commented 1 year ago

The requirement to use NGINX proxy is a step backward and a security issue. The custom integration was able to expose a specific port for a specific purpose and not expose 8123 over HTTP. Even local I do not consider exposing my entire HA instance over HTTP acceptable.

This integration should be reworked to support HTTP without exposing all of HA via HTTP like the HACS version did.

JohnMcLear commented 1 year ago

This integration should be reworked to support HTTP without exposing all of HA via HTTP like the HACS version did.

Is this accurate @veilofsecurity - It doesn't change your point but I think the above statement includes mistakes? I think it's meant to say

1) "This integration should be reworked to support HTTP for a specific purpose without the requirement to expose all of HA".

and/or (and this is for my use case).

2) "This integration should be reworked to support ecowitt where HA is exposed via core HTTPS handlers and not require Nginx reverse proxying.

They are two separate approaches. You'd of thought 2) would be supported "out of the box" given that it was implemented for hass.io.

veilofsecurity commented 1 year ago

My wording about HACS was definitely ambiguous and your 1 clarifies the meaning. Your 2 wont work because the Ecowitt gateways only support HTTP and I doubt they will be updated anytime soon, if ever.

There are 3 options:

  1. Best - Ecowitt updates to support HTTPS and we use the native integration as is.
  2. Better - HA makes the integration expose an HTTP port whose only function is to ingest from Ecowitt. This is what HACS did and doesnt make all of HA available via HTTP which is unacceptable in my opinion.
  3. Not great - Expose HA via HTTP allowing Ecowitt to send to the integration. Use NGINX Rev Proxy to enable HTTPS for HA for your everyday use, mobile apps, etc.

For now I am continuing to use the HACS integration with the line 21 code fix but I dont expect that integration to work forever as it seems to now be unmaintained. I would like to see option 2 implemented in the official integration or I am open to othet ideas if I've missed something.

adynis commented 1 year ago

Where could I vote for option "1. Best" from previous post from @veilofsecurity ? :-)

JohnMcLear commented 1 year ago

For now I am continuing to use the HACS integration with the line 21 code fix

Given the situation I might be forced to do the same ;( Can you please link to the line 21 code fix?

adynis commented 1 year ago

I suppose it's this one: https://github.com/garbled1/homeassistant_ecowitt/issues/149#issuecomment-1535912016

Cellivar commented 1 year ago

Skimming through the code it looks like the port for the webhook could be made configurable? https://github.com/home-assistant/core/blob/3fbc026d5a3dbee78778ed42548a0c2d9c0d026c/homeassistant/components/ecowitt/config_flow.py#L46

My GW1000 seems to support adding different ports, just not using HTTPS over whatever port is selected.

Entropy512 commented 1 year ago

Given that option 1 is extremely unlikely to happen (since that would require ecowitt to add the function and this does not seem likely), option 2 would make most sense. It also is the easiest for existing ecowitt users to migrate to.

Funny thing is that when installing the new Ecowitt integration after deleting all of the old HACS-originated stuff, it talks about finding an open port and even suggest 4199 which is what the old HACS ecowitt integration used - but it obviously does not actually support listening on 4199...

raucao commented 1 year ago

Just ran into this issue with a new weather station. Option 2 seems the most reasonable to me as well, because I also don't want to expose the rest of HA on HTTP. And option 1 is out of our control.

andcastellani commented 1 year ago

Is it not possible meanwhile a workaround, like writing a different port number in a config file? Or, for instance, can core/homeassistant/components/ecowitt/config_flow.py code file be edited for this type of "hacking"?

kbx81 commented 1 year ago

Just ran into this issue with a new weather station.

Add me to the list 🙃

Option 2 seems the most reasonable to me as well, because I also don't want to expose the rest of HA on HTTP. And option 1 is out of our control.

Solid agree. 😄

del13r commented 1 year ago

I also recently struggled with setting up simultaneous:

I made a guide on how I solved my issue if anyone else is currently lost: https://community.home-assistant.io/t/nginx-tls-proxy-add-on-config-to-allow-simultaneous-https-and-http-access-required-by-ecowitt-integration/589276

andcastellani commented 1 year ago

Thank you very much @del13r for your excellent guide, very useful to make ecowitt devices working again with home assistant (hoping that in the next future the integration will also allow us to choose the http port number listening for ecowitt messages)

I also recently struggled with setting up simultaneous:

  • HTTPS secure web browser access via NGINX proxy
  • HTTP webhook access direct to home assistant

I made a guide on how I solved my issue if anyone else is currently lost: https://community.home-assistant.io/t/nginx-tls-proxy-add-on-config-to-allow-simultaneous-https-and-http-access-required-by-ecowitt-integration/589276

JohnMcLear commented 1 year ago

@del13r Can you please confirm if your solution will work for Home Assistant OS users IE users who purchased Home Assistant Blue hardware and have a managed operating system?

del13r commented 1 year ago

My solution uses 2 addons. For that reason, if you are running Home Assistant Operating System, you WILL be able to install the addons. If you are running Home Assistant Container, you will NOT be able to install addons.

On this page https://www.home-assistant.io/installation it says:

If you are using the Home Assistant Blue, the Home Assistant Operating System is already installed.

I am confident the solution should work for you as well.

veilofsecurity commented 1 year ago

Skimming through the code it looks like the port for the webhook could be made configurable?

https://github.com/home-assistant/core/blob/3fbc026d5a3dbee78778ed42548a0c2d9c0d026c/homeassistant/components/ecowitt/config_flow.py#L46

My GW1000 seems to support adding different ports, just not using HTTPS over whatever port is selected.

This appears to just be the text at the end of setting up the integration and not the actual port used. Making the port selectable and listening on it would be a bit more involved than just modifying that section. Not to say it can't be done, the HACS integration managed to do it without issue.

In the meantime, I have forked the HACS integration and fixed the bug if anybody wants it: https://github.com/veilofsecurity/homeassistant_ecowitt

In my opinion this is a much better option than doing as @del13r or the official HA docs suggest. Exposing HA on HTTP is just a bad security practice.

del13r commented 1 year ago

Exposing HA on 80 is just a bad security practice.

Home assistant http default port is 8123. Nginx https proxy default port is 443.

both of these can be changed to whatever port numbers you like and you only need to publicly expose / port forward the https port and just use the http port internally without exposing/forwarding it.

JohnMcLear commented 1 year ago

@del13r RE HAOS

For that reason, if you are running Home Assistant Operating System, you WILL be able to install the addons.

Home Assistant OS supports HTTPS without the requirement of Nginx ergo exposing HA on 443. That's the documented/supported way to do it. See https://www.home-assistant.io/integrations/http/

Using nginx to reverse proxy SSL might be fine(FWIW I use it at lot but not w/ HA) but it's not the core method of supporting HTTPS and as such your documentation should reflect that a user using your documentation may have a degraded user experience or difficulties down the line. A simple warning caveat should suffice.

Also if a user already has HTTPS setup the correct way (as per HA docs) then your workaround wont work.

I do appreciate your contribution and solution though, thanks for sharing!

del13r commented 1 year ago

it's not the core method of supporting HTTPS

I see your point and I will try to make an ammendment.

In my article, I had explained that when I started, I already had HTTPS exclusively setup on default port 8123 using the default instructions for DuckDNS. I also explain that by following the default DuckDNS instructions where "you'll need to configure the Home Assistant Core to pick up the SSL certificates", that this is what upgrades default port 8123 from using http to https. Perhaps I could make that more clear.

Instuctions from https://github.com/home-assistant/addons/blob/master/duckdns/DOCS.md

Additionally, you'll need to configure the Home Assistant Core to pick up the SSL certificates. This is done by setting the following configuration for the HTTP integration configuration in your configuration.yaml:

http:
  ssl_certificate: /ssl/fullchain.pem
  ssl_key: /ssl/privkey.pem

Also if a user already has HTTPS setup the correct way (as per HA docs) then your workaround wont work.

This is why I start the article by:

http:
  ssl_certificate: /ssl/fullchain.pem
  ssl_key: /ssl/privkey.pem

I do appreciate your contribution and solution though, thanks for sharing!

Thanks for the recognition :)

veilofsecurity commented 1 year ago

Exposing HA on 80 is just a bad security practice.

Home assistant http default port is 8123. Nginx https proxy default port is 443.

both of these can be changed to whatever port numbers you like and you only need to publicly expose / port forward the https port and just use the http port internally without exposing/forwarding it.

This is correct. I meant "Exposing HA on HTTP is just a bad security practice" regardless of port used. The point still stands though, even using HTTP internally is not a good idea. You can never assume your internal environment is free of threats. I have updated my comment to avoid confusion.

del13r commented 1 year ago

even using HTTP internally is not a good idea

I agree, hence why I originally exclusively used only HTTPS on port 8123.

The problem begins and ends with the ecowitt device only supporting http for sending webhook data. This has a knock on effect which causes the native ecowitt integration to require that I expose my home assistant core using http so it can receive the http webhook data from the ecowitt device.

If you want to receive data from the ecowitt device using the native integration, this is where you have to make some choices:

I chose to do both.

To be honest, I haven't evaluated or even looked at any of the HACS alternatives yet, but I assume they will function differently to the native ecowitt integration.

raucao commented 1 year ago

The HACS one simply opens a new listener for http on a different port, only to receive this specific webhook. Which solves this problem elegantly and without drawbacks. It's the best native integration path, because it neither requires exposing the rest of hass via http, nor to run a custom reverse proxy in front of it.

del13r commented 1 year ago

You make a fair point. I only got my ecowitt device last week. I thought I would get the native integration “working” first and decided to document my journey to help me and others understand what to do make the native integration work. I may spend some time evaluating HACS options in the future

JohnMcLear commented 1 year ago

@raucao

Which solves this problem elegantly and without drawbacks.

I'm in agreement with the caveat that the drawback is that you are opening up an additional TCP port (and potentially another attack vector) and also that it's possible when you replace your router you miss adding this port forward. I'm nitpicking though ;)

Most core integrations don't open a new TCP port so it sort of makes sense that HA core wanted to do it this way, ultimately Ecowitt are the party here who need to get their act together and support HTTPS/SSL.

Has anyone actually reached out to Ecowitt for comment?

ChrisRomp commented 1 year ago

If it's helpful to anyone else, I've written a very simple Home Assistant add-on which will listen on HTTP and forward the request to the Ecowitt webhook. This hasn't been tested super thoroughly, just in my simple configuration. Input welcome.

https://github.com/ChrisRomp/addon-ecowitt-proxy

ascheucher commented 1 year ago

@ChrisRomp Thanks for sharing your little add on!

But couldn't get it up and running sadly. I created an issue in the repo.

ascheucher commented 1 year ago

With a little hint from @ChrisRomp it works now.

If somebody else wants to give it a try: But I can't get it to work. This is how I have installed it:

ChrisRomp commented 1 year ago

Definitely still some bugs to work out. As I mentioned in the other repo, I'm wondering if this would be better presented as a generic insecure webhook proxy for Home Assistant, but I'll need to understand the security implications, first, since it uses the supervisor token to talk to HA (unauthorized error without it, even though webhooks don't require auth externally).

gwatuk commented 1 year ago

Thanks for this @ChrisRomp !!! I'm not sure that the complicated installation description by @ascheucher above is necessary. I installed the Ecowitt Proxy add-on and then just did the normal Ecowitt integration installation and everything is OK.

ChrisRomp commented 1 year ago

To be fair, the documentation is a recent addition to the add on. 😅

rdperkins commented 1 year ago

Thanks, but I found my problem. I went back to the documentation on the Ecowitt integration, and after reading the following twice “Path has to match! If using the Ecowitt App, remove the first forward slash from the API token, as the app will prepend one.” I went back into the app and removed the first forward slash from the api address, and I started getting data from the Ecowitt thru Alexa, and the integration now had a device and entities. Yeah! BTW, I also have the addon-ecowitt-proxy installed.

Vicio88 commented 12 months ago

Ragazzi io ho nabu casa e l'indirizzo è https e non funziona.

Hazuno commented 11 months ago

Skimming through the code it looks like the port for the webhook could be made configurable? https://github.com/home-assistant/core/blob/3fbc026d5a3dbee78778ed42548a0c2d9c0d026c/homeassistant/components/ecowitt/config_flow.py#L46

My GW1000 seems to support adding different ports, just not using HTTPS over whatever port is selected.

This appears to just be the text at the end of setting up the integration and not the actual port used. Making the port selectable and listening on it would be a bit more involved than just modifying that section. Not to say it can't be done, the HACS integration managed to do it without issue.

In the meantime, I have forked the HACS integration and fixed the bug if anybody wants it: https://github.com/veilofsecurity/homeassistant_ecowitt

In my opinion this is a much better option than doing as @del13r or the official HA docs suggest. Exposing HA on HTTP is just a bad security practice.

Muchas gracias [veilofsecurity] por tu aportación. Gracias a tu nuevo fork, he conseguido que vuelva a funcionar Ecowitt. Actualicé después de casi un año a la versión 2023.10.5 y lógicamente había dejado de funcionar. Intenté instalar la versión integrada de Ecowitt eliminando previamente la de HACS pero no funcionaba de ninguna manera, ni siquiera instalando el proxy para Ecowitt. He instalado tu fork y todo vuelve a funcionar. Para los que no sepan como hacerlo estos son los pasos:

Perdón por escribir en español, mi inglés no es muy bueno y además también sirve de ayuda para la gente de habla hispana.

Muchas gracias de nuevo [veilofsecurity]

Coder84619 commented 9 months ago

If it's helpful to anyone else, I've written a very simple Home Assistant add-on which will listen on HTTP and forward the request to the Ecowitt webhook. This hasn't been tested super thoroughly, just in my simple configuration. Input welcome.

https://github.com/ChrisRomp/addon-ecowitt-proxy

HA 2023.12.x can't add your repo:

Cmd('git') failed due to: exit code(128) cmdline: git clone -v --recursive --depth=1 --shallow-submodules -- https://github.com/ChrisRomp/addon-ecowitt-proxy /data/addons/git/7dd6dd2c stderr: 'Cloning into '/data/addons/git/7dd6dd2c'... fatal: protocol ' https' is not supported '

Tried HTTP too, and also broken.

ChrisRomp commented 9 months ago

(Sharing for others in case they have a similar issue.) @Coder84619 likely an issue with a leading whitespace in your URL per my notes in the issue: https://github.com/ChrisRomp/addon-ecowitt-proxy/issues/19

ChrisRomp commented 9 months ago

There's also a "My" button on the home page of the add-on repository to add it to your HA instance:

... or regular URL if you prefer ... https://my.home-assistant.io/redirect/supervisor_add_addon_repository/?repository_url=https%3A%2F%2Fgithub.com%2Fchrisromp%2Faddon-ecowitt-proxy

Coder84619 commented 9 months ago

Here's a blog post that covers the full proxy add-on configuration: https://www.derekseaman.com/2023/12/home-assistant-ecowitt-weather-station-setup.html

rdperkins commented 9 months ago

One of the recent HA updates resolved the Ecowitt problem – at least for my rain gauge. Thanks.

From: Coder84619 @.> Sent: Saturday, December 23, 2023 3:26 PM To: home-assistant/core @.> Cc: Robert Perkins @.>; Comment @.> Subject: Re: [home-assistant/core] Ecowitt Integration - HTTPS requirements non functional (Issue #93000)

Here's a blog post that covers the full proxy add-on configuration: https://www.derekseaman.com/2023/12/home-assistant-ecowitt-weather-station-setup.html

— Reply to this email directly, view it on GitHubhttps://github.com/home-assistant/core/issues/93000#issuecomment-1868364780, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AH55NHZ24NCYL4CIT637BL3YK444PAVCNFSM6AAAAAAX7YOERCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGM3DINZYGA. You are receiving this because you commented.Message ID: @.**@.>>

rdperkins commented 9 months ago

Thanks, everything’s working with my rain gauge now. Much appreciated.

From: Chris Romp @.> Sent: Tuesday, December 19, 2023 3:54 PM To: home-assistant/core @.> Cc: Robert Perkins @.>; Comment @.> Subject: Re: [home-assistant/core] Ecowitt Integration - HTTPS requirements non functional (Issue #93000)

There's also a "Myhttps://my.home-assistant.io/faq/" button on the home page of the add-on repository to add it to your HA instance:

image.png (view on web)https://github.com/home-assistant/core/assets/1683889/719acb40-cdac-4886-8c64-cba01bbebc8e ... or regular URL if you prefer ... https://my.home-assistant.io/redirect/supervisor_add_addon_repository/?repository_url=https%3A%2F%2Fgithub.com%2Fchrisromp%2Faddon-ecowitt-proxy

— Reply to this email directly, view it on GitHubhttps://github.com/home-assistant/core/issues/93000#issuecomment-1863457319, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AH55NH53BNXYWHLQX2LY5E3YKH5F5AVCNFSM6AAAAAAX7YOERCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRTGQ2TOMZRHE. You are receiving this because you commented.Message ID: @.**@.>>

jimangel commented 9 months ago

Throwing another solution in the mix (running a simple nginx reverse proxy DIY): https://gist.github.com/jimangel/77bd9393e0097e748005fb8ee44db6bf

I wanted to own the solution 100% to ensure security / meeting my specific needs, however I think anyone willing to run nginx in a container (or otherwise) on their network would find value in the solution.

In my case, I'm running Home Assistant on an Intel NUC with a Docker / Supervisor install and do NOT want to enable a HTTP endpoint on my HA. I'm also using Nabu Casa as my external HTTPS webhook destination for ecowitt but that could be any HTTPS endpoint. Best of luck!

ChrisRomp commented 9 months ago

@jimangel Thanks for sharing that. I'd also considered it but using the Nginx add-on from the official "store." Eventually I might document that and deprecate my add-on in favor of it if I can find some time. :)

issue-triage-workflows[bot] commented 6 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

Cellivar commented 6 months ago

I don't believe any changelog has indicated this has been fixed.

del13r commented 6 months ago

I don't believe any changelog has indicated this has been fixed.

I am in the same boat as you, however there are alternative methods around this limitation. May I ask what the expectation is with this issue?

The following points may affect why this issue has not been resolved:

Example configuration.yaml entry show what users do to convert Home Assistant from HTTP to HTTPS

http:
  server_port: 443
  ssl_certificate: fullchain.pem
  ssl_key: privkey.pem

Don't get me wrong, I am also interested in a native solution that plays nicely with HTTPS, however I can understand why it may not be possible and it all comes back to the device not supporting HTTPS.

Coder84619 commented 6 months ago

This is still most certainly an issue if HA is using trusted HTTPS certificate. Ecowitt does NOT natively support HTTPS to DIY servers. This needs to be fixed on HA's end, and not by some NGINX hack or third party proxy.

veilofsecurity commented 6 months ago

We expect it to work as it did before. Expose a separate HTTP port for this specific purpose only. Other than the modified original integration that is unmaintained and could break anytime, all other integrations are either overly complex or require exposing all of HA over HTTP which is unacceptable even locally.

del13r commented 6 months ago

Expose a separate HTTP port for this specific purpose only.

For this to happen, the exposed http port would need a whitelist function with just the IP of the ecowitt gateway device on it and anything else would be considered blacklisted and blocked.

Has anyone seen any other core integrations successfully do this before?

Momro commented 5 months ago

I highly appreciate everyone's work on this issue and the discussion. I find it extremely irritating that ecowitt does not allow https, not even with a publicly trusted certificate. Is this 2003 or what?

However, I don't understand why HA does in fact not offer any way to access the system with http AND https, like most Web servers do (but most will forward http to https, which could be configurable).

I'll try the previously mentioned ecowitt proxy solution. An additional nginx reverse proxy sounds neat, but like lots of additional work for something that simple as offer http.

jimangel commented 4 months ago

My nginx solution ultimately ended up breaking on me a few weeks ago, and I have since abandoned all Ecowitt products; which is a shame given the amount of investment I had in the ecosystem.

The platform I replaced it with works great and everything just works with Home Assistant; as it should. I am now in electronic-smart-home-weather bliss.

I'd share the replacement info but don't want to sound like an advertisement. For now, I'd like to focus on the shortcomings of Ecowitt 😅 and I hope they update their app!