Web-UI page empty on 0.101.0

[jrhbcn]

Hello,

I just updated to latest version of the add-on (0.101.0) together with latest versions of home-assistant (0.114.1) and supervisor (234). I am running Home Assistant Supervised in a raspberrypi4 running raspbian. Everything was working properly before updating. However now I can not access the web-ui of the add-on and there seems to be nothing in the logs (with verbose: true):

08-15 15:37:39 INFO Starting server on port 8099
08-15 15:37:39 INFO Server started
08-15 15:37:39 INFO Syncing Snapshots

However there are some errors in the Supervisor log (but I don't know if they are related):

20-08-15 13:37:34 ERROR (MainThread) [supervisor.api.ingress] Ingress error: 
20-08-15 13:37:34 ERROR (MainThread) [supervisor.api.ingress] Ingress error: 
20-08-15 13:37:35 INFO (SyncWorker_5) [supervisor.docker.interface] Clean addon_cebe7a76_hassio_google_drive_backup application
20-08-15 13:37:37 INFO (SyncWorker_2) [supervisor.docker.addon] Start Docker add-on sabeechen/hassio-google-drive-backup-armv7 with version 3.12
20-08-15 13:37:39 INFO (MainThread) [supervisor.api.security] /homeassistant/info access from cebe7a76_hassio_google_drive_backup
20-08-15 13:37:39 INFO (MainThread) [supervisor.api.security] /supervisor/info access from cebe7a76_hassio_google_drive_backup
20-08-15 13:37:39 INFO (MainThread) [supervisor.api.security] /snapshots access from cebe7a76_hassio_google_drive_backup
20-08-15 13:37:39 INFO (MainThread) [supervisor.api.security] /snapshots/7876c97e/info access from cebe7a76_hassio_google_drive_backup
20-08-15 13:37:39 INFO (MainThread) [supervisor.api.security] /snapshots/cc78f0f7/info access from cebe7a76_hassio_google_drive_backup
20-08-15 13:37:39 INFO (MainThread) [supervisor.api.security] /snapshots/24a61829/info access from cebe7a76_hassio_google_drive_backup
20-08-15 13:37:39 INFO (MainThread) [supervisor.api.security] /snapshots/4a31f353/info access from cebe7a76_hassio_google_drive_backup
20-08-15 13:37:39 INFO (MainThread) [supervisor.api.security] /snapshots/bca5966e/info access from cebe7a76_hassio_google_drive_backup

[sabeechen]

Definitely related. Ingress is what the supervisor calls the technique it uses to embed an addon's web UI inside the Home Assistant interface. I suspect this is a problem with the supervisor, but that log message doesn't give a whole lot of help. From the supervisor's source code it looks like it should be printing out spitting out some other details but strangely none are showing up in your supervisor log. I'd recommend either:

• Restart the supervisor (or the machine you run HA on). Honestly, this solves almost every problem with the supervisor.
• Reinstall the addon. This solves almost every supervisor problem that restarting doesn't :). If you do this, make sure you backup the addon config somewhere and the copy it back in before starting the addon again.
• You can add the config: expose_extra_server: True To have the addon additionally serve a web-ui on port 1627. You should then be able to reach the web interface via http://homeassistant:1627/ (or whatever the hostname for your Home Assistant is). This would verify if the problem is with the addon or not.

[jrhbcn]

Thanks for the fast answer!

• Restart the supervisor (or the machine you run HA on). Honestly, this solves almost every problem with the supervisor.

Done. Nothing changed.

• Reinstall the addon. This solves almost every supervisor problem that restarting doesn't :). If you do this, make sure you backup the addon config somewhere and the copy it back in before starting the addon again.

Done, still same problem.

• You can add the config: expose_extra_server: True To have the addon additionally serve a web-ui on port 1627. You should then be able to reach the web interface via http://homeassistant:1627/ (or whatever the hostname for your Home Assistant is). This would verify if the problem is with the addon or not.

It works there, so definitely something going on about the supervisor. In any case, I am going to re-install all my home-assistant to boot from USB so maybe then it'll work :) I'll let you know...

[sabeechen]

Best of luck!

[Appyx]

Hi, I have the same problem on supervisor 234 and HA 0.114.1.

The addon works with the secondary webserver but not the ingress part.

Things I tried:

• Restart Addon
• Reinstall Addon
• Reload Supervisor
• Reboot Host

I'd really like to use the handy ingress feature 😄

[jrhbcn]

Robert, just to see if we can find any coincidence with me. What hardware are you using? Which installation method of home assistant?

[Appyx]

I'm using RPI4 4gb with the 32bit image (native installation).

Edit: Just upgraded to 114.2 and no luck with it.

I also remember a log message when I installed the addon the first time: "The Supervisor rejected the addon"....or something like this.

[DasLetzteEinhorn]

Same error here. With the option expose_extra_server: true enabled, I can access the WebUI flawlessly - only via ingress the page is empty.

[Appyx]

Looks like the iframe is loaded into the ingress-view but the body is not built.

Edit:

After a little while the body is loaded following two 502 status code responses when loading the following resource: /static/css/colpick.css?version=0.101.0

And when it is loaded, then the styling looks a little skewed 🤔

It also periodically looses connection to HA...again 502 status code when loading /getstatus.

So looks like a general "Bad Gateway" thing.

[jrhbcn]

I also see a weird thing that after trying to open the web-ui of the addon (which is blank) and then go back to the supervisor iframe it also does not display ... and I have to wait for a while for the supervisor ui to load. Does it happen to you also?

[sabeechen]

@Appyx thansk for posting that, that gives me more to work with. In my experience, the supervisor's proxy returns a 502 error for basically any error the addon's webserver returns. Its good to know something is getting through. I'd guess the delay in loading the body is just your browser attempting to load all the stylesheets and javascript resources before it attempts to actually render the page.

• /static/css/colpick.css is just one of the addon's many stylesheets (used for the settings color-picker), it isn't served any differently than any of the other static resources fromthe addon.
• /getstatus is what the addon calls periodically to refresh the web-UI while you're watching it. It should request once every 5 seconds or so when things are working right.

Its hard to guess why some requests might get through and others would fail, all the endpoints for the addon are served in more-or-less the same way. It kind of sounds like the supervisor is getting hung up randomly, so parts of the web-ui loads paritally or not at all, which is why you'd see things styled incorrectly. Since many people have reported the web-ui working when exposed directly on a port, I'm still inclined to think this is a problem with the supervisor.

Come to think of it, I've been having quite a few problems with the latest supervisor version. Getting kind of fed up with the supervisor, actually. But maybe its something I'm doing.

@Appyx Do you see anything in the supervisor logs when the UI completes "partially" loading (eg at Supervisor > System)? The original user who posted saw a mostly blank error message [supervisor.api.ingress] Ingress error: but maybe something helpful gets printed out when you let it try to complete?

Some extra context for those who might be curious: The supervisor handles "ingress" (embedded addons like this one) by proxying requests from your browser to the web server's addon. So for example this addon exposes a web server (inside a docker container), and the supervisor makes it available by loading the addon in an iframe. This is how all embedded addons work.

[sabeechen]

I wish I'd thought of this before, but could someone encountering this see if the web ui loads after disabling popup-blockers, ad-blockers, etc? Alternatively just try opening it in a different browser all together (edge, firefox, etc)

[DasLetzteEinhorn]

I wish I'd thought of this before, but could someone encountering this see if the web ui loads after disabling popup-blockers, ad-blockers, etc? Alternatively just try opening it in a different browser all together (edge, firefox, etc)

yup, just tried it - still the same, sorry.

[Appyx]

@sabeechen I tried what you suggested and the only logs i get is:

20-08-19 19:36:19 ERROR (MainThread) [supervisor.api.ingress] Ingress error:

I even upgraded to a HA OS 5.1 64 bit image booted from SSD. And it's the same.

During the upgrade i found another log (might be not relevant though):

20-08-19 19:23:10 WARNING (SyncWorker_3) [supervisor.docker.interface] Can't find sabeechen/hassio-google-drive-backup-aarch64 for cleanup

[Appyx]

@sabeechen I think I solved it.

When I access HA via http://homeassistant.local:8123 (which is essentially http://10.0.0.200:8123), then the ingress works. When I access HA via my proxy https://***.house (which proxies to http://10.0.0.200:8123), then it doesn't.

However it might still be an issue with the addon because all other addons work via my proxy 🤔

[sabeechen]

Yup, that sounds like I'm probably doing something wrong. Though strange that it only has problems with certain requests and not all of them, hopefully everyone is seeing the same issue. The latest release of the addon uses a different webserver, and I'm guessing it doesn't play nice with your configuration.

Would you be willing to describe how your proxy is set up? Obviously not detailed addresses and stuff, just what I'd need to reproduce it. I'd like to try to replicate your setup to see if I can figure out where things are going bad.

[Appyx]

I'm using the Nginx Proxy Manager Addon mapping https://domain to http://10.0.0.200:8123. The ssl is done using the built in LetsEncrypt. So my router forwards everything with port 80 and 443 to 10.0.0.200. I also use split dns to resolve the domain to a local address when inside my network. This is also done on the router.

I hope this helps.

[jrhbcn]

I am also using nginx to proxy https requests. In my case installed locally in another raspbian machine. The relevant config is this one (i can not tested it right now as I am in the process of transfering my Home Assistant installation to boot from SSD directly using hassos):

server {
    listen       443 ssl;
    server_name  xxxx.duckdns.org;

    ssl on;
    ssl_certificate /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/privkey.pem;
    ssl_trusted_certificate /etc/nginx/certs/chain.pem;
    ssl_prefer_server_ciphers on;

    location / {
        proxy_pass http://192.168.1.200:8123;
        proxy_set_header Host $host;

        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

[DasLetzteEinhorn]

This is the page I see sometimes - but no button works. I also takes about one minute to load:

[sabeechen]

I've tried to reproduce this on a number of platforms and configurations (pi2 32 bit, pi3 64 bit, pi4 64 bit, 64 bit ubuntu, using the nginx proxy manager addon, and using a custom nginx reverse proxy) but haven't been able to get the problem to happen so I'll need some more help.

I've released a new version of the addon (0.101.2) thats capable of providing a lot more logging output that might help figure out whats going on here. Could I have you guys do the following:

• Make sure you're using the latest version of the supervisor and update if necessary
• Update the addon to 0.101.2
• Add the following to your addon config:
  • log_level: 'TRACE'
  • console_log_level': 'TRACE'
  • expose_extra_server: true This will make the addon produce an obscene amount of logs, and buried in there is hopefully the info I need. These options cant be set using the web-ui.
• Restart the addon, visit web-ui over ingress (eg via the sidebar) and wait for it to fail to load.
• Navigate to http://homeassistant.local:1627/log?format=view. At the top should be a link to download the addon's logs, click it and save them somewhere. Your local address for home assistant might be different but the port should be 1627.
• Attach the logs here or email them to me stephen@beechens.com if you're worried about leaking personal information. I've taken some care to ensure no sensitive info gets printed, but I can't be certain.

[jrhbcn]

OK, I have managed to restore my HA system booting directly from SSD. Updated the addon to 0.101.0 and I am experiencing the same problem :( . I have email you the log directly. Many thanks for your efforts to solve this, let's see if we can find the problem!!

[jrhbcn]

Btw, I am still not 100% convinced this issue is isolated from the supervisor 235 version. :? As I commented earlier, at least on my setup, once I try to open the addon through the web-ui window (and failing) it then fails to load the normal supervisor window on home assistant. It seems as if I need to wait for a bit, click somewhere else or refresh and then supervisor window works....

[darox]

I'm also having this issue on supervised (235) ha version 0.113.3.

[PeteBauer]

I'm also experiencing this issue on superviser 235 and HA 0.114.4 Using the additional web gui on port 1627 gives no issues

I'm also using a proxymanager (NGINX)

[sabeechen]

Based on some of the logs I've been getting back, here is what the order and timing of requests looks like when the web-ui fails to load.

  GET /
  GET /theme
  GET /static/js/materialize.js
  GET /static/js/init.js
  GET /static/js/scripts.js
(1 minute later)
  GET /static/images/logo.png
(1 minute)
  GET /static/js/restore.js
(19 seconds later)
  GET /theme

In the logs I can see that the addon server responds to requests almost immediately for each one. The long delays seem to be coming from the browser (or proxy). I don't see any errors in the logs either for the requests that come through, it looks a lot like the the requests that fail or timeout aren't ever making it to the addon's server at all. For reference here is what the pattern of requests looks like for a healthy server:

    GET / 
    GET /theme
    GET /static/css/icons.css
    GET /static/js/jquery.js
    GET /static/css/materialize.css
    GET /static/css/static.css
    GET /static/css/colpick.css
    GET /static/js/markdown-it.min.js
    GET /static/js/materialize.js
    GET /static/js/colpick.js
    GET /static/js/init.js
    GET /static/js/settings.js
    GET /static/js/scripts.js
    GET /static/js/restore.js
    GET /static/js/moment.js
    GET /static/images/logo.png
    GET /static/images/bmc.svg
    GET /static/css/iconfont/MaterialIcons-Regular.woff2
    GET /static/images/favicon.png   

In the broken request pattern it didn't even attempt to load a number of resources (eg jquery.js) that are required to for everything else to work and render properly. The pattern of the gaps between requests is also suspicious, exactly one minute. Waiting exactly one minute looks a lot like a timeout somewhere expiring, but nothing in the addon (that I'm aware of) works on a 1 minute timeout.

this looks a lot to me like something is either wrong in browser, or in the proxy. Since the extra server on port 1627 seem to be working fine, I'm really leaning toward something being wrong with the proxy. This could be either the supervisor's proxy (eg to make ingress work) or the external proxy.

Debugging this without being able to reproduce it is proving difficult.

If i can't reproduce this on one of my machines, then I won't be able to fix it unless one of you seeing it can figure out whats wrong :(

Could I have someone share with me their exact proxy configuration file with me including the proxy software name and version?

[jrhbcn]

Hi Stephen,

In my case I am using Home Assistant OS 64 bits running on a raspberry pi 4 (installed directly on SSD but it also happended using normal SD installation and 32 bits). I have another raspberry pi 4 machine running latest raspbian configured with nginx to do the proxy redirection to the Home Assistant machine.

nginx is installed from raspbian packages. Current version is:

pi@raspberrypi4: $ dpkg -l nginx
ii  nginx          1.14.2-2+deb10u3 all          small, powerful, scalable web/proxy server

And it is configured as:

server {
    listen       443 ssl;
    server_name  xxxx.duckdns.org;

    ssl on;
    ssl_certificate /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/privkey.pem;
    ssl_trusted_certificate /etc/nginx/certs/chain.pem;
    ssl_prefer_server_ciphers on;

    location / {
        proxy_pass http://192.168.1.200:8123;
        proxy_set_header Host $host;

        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

Just to make another test, I changed the nginx configuration to directly go to http://192.168.1.200:1627 (before that I was going directly through local network) and it works fine. Don't know how different the addon web server is between port 1627 and directly through web-ui though....

Let me know if you need any more tests from my setup!

[PeteBauer]

If got some more information about this. 3 colleagues of mine are also using the google backup addon, but with different configs: 1: uses hassos, with nginx proxymanager (NGINX Home Assistant SSL proxy) addon, with duckdns addon. No use of Docker. He has no issues starting the backup addon. 2: uses raspbian, without a proxymanger. Uses Docker. He has no issues starting the addon 3: uses a NUC, with nginx proxymanager (NGINX Home Assistant SSL proxy) addon, with duckdns addon. Uses Docker. He has no issues starting the backup addon

My setup is: raspbian, with Nginx Proxy Manager addon (without DuckDNS addon). I'm using Docker. I have issues starting the backup addon

EDIT: as of Aug 31 2p.m. I don't have issues anymore with starting the backup addon....

[mgriffin13]

Hi, I've been following this issue since I'm experiencing the same thing. What I've found:

1. If I connect to http://hassio.local:1627, the UI loads perfectly.
2. If I connect to http://hassio.local:8123, the UI loads perfectly through Ingress
3. If I connect to https://$myhassiodns, through my nginx reverse proxy, the UI fails to load.

I'm running my nginx reverse proxy on a completely different host, so I'm not using the NGINX Home Assistant SSL proxy. My relevant nginx configuration

# main server block
server {
    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;

    server_name _;

    # enable subfolder method reverse proxy confs
    include /config/nginx/proxy-confs/*.subfolder.conf;

    # all ssl related config moved to ssl.conf
    include /config/nginx/ssl.conf;

    # enable for ldap auth
    #include /config/nginx/ldap.conf;

    client_max_body_size 0;

    location / {
        proxy_set_header Host $host;
        proxy_redirect http:// https://;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_buffering               off;
        proxy_ssl_verify              off;
        proxy_pass http://192.168.0.86:8123;
    }
}

# enable subdomain method reverse proxy confs
include /config/nginx/proxy-confs/*.subdomain.conf;
# enable proxy cache for auth
proxy_cache_path cache/ keys_zone=auth_cache:10m;

[sabeechen]

I've made some progress, though I don't have things fully figured out yet. This has something to do with websockets and connection upgrades getting passed through the proxies.

Special thanks to @jrhbcn and @mgriffin13, I was able to set up a host with all the necessary proxy routing to reproduce the problem. Finally!

The web-ui hangs on this minimal Nginx config:

server {
    listen 80;

    location / {
        proxy_set_header Host $host;
        proxy_redirect http:// https://;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass http://192.168.0.100:8123;
    }
}

But it works properly with this config:

  map $http_upgrade $connection_upgrade {
      default upgrade;
      ''      close;
  }

  server {
    listen 80;

    location / {
        proxy_set_header Host $host;
        proxy_redirect http:// https://;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_pass http://192.168.0.100:8123;
    }
}

The critical difference is the line proxy_set_header Connection $connection_upgrade; and the map defined up top that defines $connection_upgrade. I was able to find some Nginx documentation that describes this, but I'm still not sure what I might be doing wrong with the addon's web server. This problem only started happening when I switched the addon's webserver to serving through the AioHttp server library, so clearly that is related.

Other addons seem to render fine without the change above, and moreover the web-ui also loads fine with the "broken" config above if I proxy it directly to the addon's webserver (eg port 1627). This issue only comes up when requests get routed through: Nginx proxy -> Supervisor ingress proxy -> Addon webserver

I'll continue to try to figure this out, but I'm digging through a domain of knowledge I know nothing about. If someone understands what this Connection header does or why it might be causing problems, please chime in and help.

For the time being adding the map and proxy_set_header Connection $connection_upgrade; line seems to be a fix. For my config I copied it from this documentation on proxying home assistant.

[mgriffin13]

Yep! I made that change by adding the map and it's the snapshot ui is working again through the nginx proxy. Thanks!

[jrhbcn]

Same here, adding the map option in the nginx config solved the problem completely! I didn't use the _use_x_forwardedfor and _trustedproxies in the http section of the home assistant configuration though (as suggested in the proxy documentation).

I also not able to help much about why it works with these nginx config changes :? In any case, many thanks again for looking into it and solving it!

Regards,

[sabeechen]

I think I'm going to call this bug resolved since it appears to be a proxy configuration issue more then anything else. If I missed anything, please feel free to reopen this and I'll take a crack at it.

[omriasta]

I added this to my nginx reverse proxy but still have the same issue, web ui looks like the attached, everything else in homeassistant is working fine except for this addon and here is my nginx config:

map $http_upgrade $connection_upgrade {
      default upgrade;
      ''      close;
  }
server {
        listen 443 ssl;

        root /config/www;
        index index.html index.htm index.php;

        server_name xxxxxx.duckdns.org;

        include /config/nginx/ssl.conf;

        client_max_body_size 0;

        location / {
                proxy_set_header Host $host;
                proxy_redirect http:// https://;
                proxy_http_version 1.1;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "upgrade";
                proxy_buffering               off;
                proxy_ssl_verify              off;
                proxy_pass http://x.x.x.x:8123;
        }
location /api/notify.html5/callback {
    if ($http_authorization = "") { return 403; }
    allow all;
    proxy_pass http://x.x.x.x:8123;
    proxy_set_header Host $host;
    proxy_redirect http:// https://;
}
}

[sabeechen]

You need to add:

proxy_set_header Connection $connection_upgrade;

to your location / block and remove the existing proxy_set_header Connection line.

[omriasta]

That was it, sorry must have misread it when I was trying to add this in :)