Unknown error occurred / 504 timeout - Error in IPMItool server when adding the IPMI Connector integration to Home Assistant

myklebosten commented 11 months ago

Thanks for the integration and add-on!

I seem to be runing into a bug of some kind while adding the IPMI Connector to HA.

System details

I have installed both the HACS integration IPMI Connector and the Add-on IPMItool server.

My setup

Add-on: IPMItool server IPMItool server ----------------------------------------------------------- Add-on version: 1.1.25 You are running the latest version of this add-on. System: Home Assistant OS 11.1 (amd64 / qemux86-64) Home Assistant Core: 2023.11.2 Home Assistant Supervisor: 2023.11.3 ----------------------------------------------------------- All supplied logs originates from my HA OS running in a VM. Also tested on dedicated hardware (HA Yellow) with same results.

IPMItool Server startup

```shell s6-rc: info: service base-addon-banner successfully started s6-rc: info: service fix-attrs: starting s6-rc: info: service base-addon-log-level: starting s6-rc: info: service fix-attrs successfully started Log level is set to DEBUG s6-rc: info: service base-addon-log-level successfully started s6-rc: info: service legacy-cont-init: starting s6-rc: info: service legacy-cont-init successfully started s6-rc: info: service legacy-services: starting services-up: info: copying legacy longrun nginx (no readiness notification) services-up: info: copying legacy longrun php-fpm (no readiness notification) [16:07:31] INFO: Starting PHP-FPM... s6-rc: info: service legacy-services successfully started [16:07:32] INFO: Starting NGinx... ```

Problem

When adding the IPMI connector integration with known working connection details, I get an error:

(!) Unknown error occurred

**IPMItool server Logs** ```shell 2023/11/16 11:27:32 [error] 205#205: *17 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "my-iot-url.duckdns.org" ``` **Ingress when using the URL from the log** **URL:** https:// _my-iot-url.duckdns.org/api/hassio_ingress/3aZ8iC0mb6JKifA0UIsATgsPFg5Y5ww7pnUHdEcZX8w/?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword_ ---------------------------------------------------------- 504 Gateway Time-out nginx ----------------------------------------------------------

Further debuging

When adding the integration with with known not working connection details (like a bad password or host), i get a different error:

(!) Failed to connect

**IPMItool server Logs** ```shell 2023/11/16 11:27:32 [error] 205#205: *17 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "my-iot-url.duckdns.org" ``` **Ingress when using the URL from the log** **URL:** https:// _my-iot-url.duckdns.org/api/hassio_ingress/3aZ8iC0mb6JKifA0UIsATgsPFg5Y5ww7pnUHdEcZX8w/?host=192.168.0.52&port=623&user=MyIPMIUsername&password=MyPassword_ ```shell {"success":false,"message":"Wrong connection data provided!","debug":"Error occurred when running \u0022ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lanplus bmc info\u0022.\n Unable to Get Channel Cipher Suites\nError: Unable to establish IPMI v2 \/ RMCP+ session\n\n Error occurred when running \u0022ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lan bmc info\u0022.\nAuthentication type NONE not supported\n Error: Unable to establish LAN session\n Error: Unable to establish IPMI v1.5 \/ RMCP session\n\n Error occurred when running \u0022ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I imb bmc info\u0022.\n Error loading interface imb\n\n Error occurred when running \u0022ipmitool -H 192.168.0.52-p 623 -U MyIPMIUsername -P MyPassword -I open bmc info\u0022.\n Could not open device at \/dev\/ipmi0 or \/dev\/ipmi\/0 or \/dev\/ipmidev\/0: No such file or directory\n "} ``` (I added new lines for every \n to make it more readable)

When runing the ipmitool command from Failed to connect with the correct data from the Unknown error occurred, I get this:

Shell output:

```shell ~$ ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lanplus bmc info Device ID : [Redacted] Device Revision : [Redacted] Firmware Revision : [Redacted] IPMI Version : 2.0 Manufacturer ID : [Redacted] Manufacturer Name : [Redacted] Product ID : [Redacted] Product Name : [Redacted] Device Available : yes Provides Device SDRs : no Additional Device Support : Sensor Device SDR Repository Device SEL Device FRU Inventory Device Chassis Device ``` (Tell me if you need any of the redacted info)

Appreciate if you have time to help me debug this.

I unfortunately don't have the skils to dig further by my self. Please tell me if there is anything I can do to assist :)

ateodorescu commented 11 months ago

Where did you run the working ipmitool command from? Was that inside the docker installation of the addon or just one of your systems? Because you get a timeout I'm thinking that maybe the VM for HASS (including the docker container for the addon) has no access to the network of your server 192.168.0.52. Please look into this.

myklebosten commented 11 months ago

Thanks for the answer!

No, I ran ipmitool on my own linux host as I'm not that familiar with docker. But I read a tutorial and did connect to the docker host now. Here's the output:

root@304fecf0-ipmi-server:/$ ipmitool -H 192.168.0.52 -p 623 -U MyIPMIUsername -P MyPassword -I lanplus bmc info
Unable to Get Channel Cipher Suites
Device ID                 : [Redacted]
Device Revision           : [Redacted]
Firmware Revision         : [Redacted]
IPMI Version              : 2.0
Manufacturer ID           : [Redacted]
Manufacturer Name         : [Redacted]
Product ID                : [Redacted]
Product Name              : [Redacted]
Device Available          : yes
Provides Device SDRs      : no
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    Chassis Device

I also tested the integration and addon on dedicated hardware (Home Assistant Yellow) with same result. Wanted to be sure the VM environment wasn't the cause of the issue before posting. That info probably drowned in my way too long post yesterday.

ateodorescu commented 11 months ago

Please open in a browser this link: http://YOUR_HASS_SERVER_IP:9595/command?params=-I%20lanplus%20-H%20YOUR_IPMI_SERVER_IP%20-U%20ADMIN%20-P%20YOUR_PASSWORD%20bmc%20info and let's see the result.

myklebosten commented 11 months ago

That one worked :)

{"success":true,"output":"Device ID                 : **\n
Device Revision           : *\n
Firmware Revision         : *.**\n
IPMI Version              : 2.0\n
Manufacturer ID           : **\n
Manufacturer Name         : *\n
Product ID                : **** (0x****)\n
Product Name              : Unknown (0x****)\n
Device Available          : yes\n
Provides Device SDRs      : no\n
Additional Device Support :\n
    Sensor Device\n
    SDR Repository Device\n
    SEL Device\n
    FRU Inventory Device\n
    Chassis Device\n
"}

myklebosten commented 11 months ago

By the way, still no success on adding the integration. And when using the URL from before (though last time I did it through ingress) I still get the 504 Gateway Time-out.

ateodorescu commented 11 months ago

Well, this is strange. Are you sure there is no typo in user/password/server_ip when typing them in the integration dialog? I can't find a better reason for what is happening.

myklebosten commented 11 months ago

Agreed :)

Yes, I'm sure. I've typed it so many times now, I'm beginning to doubt myself. So this time I copied the info from the last known working command.

root@304fecf0-ipmi-server:/$ cat /var/log/php81/error.log
[16-Nov-2023 16:07:31] NOTICE: fpm is running, pid 156
[16-Nov-2023 16:07:31] NOTICE: ready to handle connections

root@304fecf0-ipmi-server:/$ cat /var/log/nginx/error.log

Any other logs I can look into?

ateodorescu commented 11 months ago

Any other logs I can look into?

None that I know of.

myklebosten commented 11 months ago

And as mentioned earlier: If I do type something wrong, I get a different error:

Integration: (!) Failed to connect Add-on url: Produces output

If I have correct connection details, the integration produces the error:

Integration: (!) Unknown error occurred Add-on url: Produces a 504 nginx error

myklebosten commented 11 months ago

I'll try to look into it a bit more when I have time later. Work (and/or lunch) is calling :)

myklebosten commented 11 months ago

I'm sorry, but I cant seem to find the problem. I've tried searching around a bit, and my guess is it's something to do with how nginx is communicating with php/fastcgi. If you want to continue, I'll do testing and stuff if needed. If not, you can close this issue :)

By the way: I would look into how to run this integration purely in ingress, with no external access. Shouldn't be too hard to do, just point your code to the internal docker host instead of an external host/IP on port 9595. That way, it won't be as easy to sniff the password over the network and get access to rebooting the server and what not.

ateodorescu commented 11 months ago

By the way: I would look into how to run this integration purely in ingress, with no external access. Shouldn't be too hard to do, just point your code to the internal docker host instead of an external host/IP on port 9595. That way, it won't be as easy to sniff the password over the network and get access to rebooting the server and what not.

I don't know how to do that. If you show me how then I could do it.

myklebosten commented 11 months ago

I'm not a HA dev, so I had to do some research for you on how to do this.

Integration and add-on today

You already have the ingress enabled in your add-on config, probably as per the docs here: https://developers.home-assistant.io/docs/add-ons/presentation/#ingress So I can enable the ipmi-tool in the sidebar of Home Assistant, and access it there (with urls and everything).

The add-on is installed with port 9595 enabled by default. This means the add-on webgui is accessible both via

ingress - with HA handling the login/access
externally on port 9595 - open access to anyone without login, and with no SSL/https support so the passwords can easily be sniffed with a simple network traffic analyzer (ipmi protocol might still send passwords in plain text though, I don't know).

Your integration (not add-on) needs the external port 9595 enabled, or it falls back to using python-ipmi. I have no integration coding nowledge, but as far as I can see line 21 in const.py is handling where to connect with the string IPMI_URL = "http://localhost". In the config flow of the integration, we can change the port.

Modifications needed

[ ] Point to add-on host instead of localhost I have little to no HA coding experience, but this looks promising: https://developers.home-assistant.io/docs/add-ons/communication/ It states that to connect to the host, you can use the name/alias, which in your case shoud be 304fecf0_ipmi-server (hostname generated as per docs in the {REPO}_{SLUG} format, so it should stay constant as long as you don't change the slug ipmi-server).
[ ] Remove the port-option from config flow As this is now handled in config which is not user changable :D
[ ] Check your add-on configuration Docs: https://developers.home-assistant.io/docs/add-ons/presentation/#ingress
- [ ] Ingress config Double check the config is correct.
- [ ] Decide if you want to use the default port 8099 or if you want to manually use another port. Just test what works, you'll figure it out :)
- [ ] Disable external access on port 9595 by default

Final words

You told me to go to an url during debuging: http://YOUR_HASS_SERVER_IP:9595/command?params=-I%20lanplus%20-H%20YOUR_IPMI_SERVER_IP%20-U%20ADMIN%20-P%20YOUR_PASSWORD%20bmc%20info.

You can do the same in ingress (if you didn't already know): http[s]://YOUR_HA_URL/304fecf0_ipmi-server/ingress/command?params=-I%20lanplus%20-H%20YOUR_IPMI_SERVER_IP%20-U%20ADMIN%20-P%20YOUR_PASSWORD%20bmc%20info

Hope this helps :)

ateodorescu commented 11 months ago

Thanks! When I find the time I will look into that.

ateodorescu commented 7 months ago

@myklebosten try out the latest addon version 1.2.7 and see if that works for you. Thanks!

ateodorescu / home-assistant-addons