Upgrade Succeed From 0.18.1-rc4&7 to 0.18.1-rc9 But Server Error?

[fmunim]

Describe the bug I upgraded using the command sudo ./deploy.sh -f -l 0.18.1-rc.9 -w 0.18.1-rc.9 and it seemed to work but when I open up the site I am greeted with:

Logs only says this:

lemmy-easy-deploy-lemmy-ui-1  | FetchError: request to https://domain/pictrs/image/dc21065f-45ac-4043-bca5-82fcd2e96545.png failed, reason: connect ECONNREFUSED 127.0.1.1:443
lemmy-easy-deploy-lemmy-ui-1  |     at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
lemmy-easy-deploy-lemmy-ui-1  |     at ClientRequest.emit (node:events:511:28)
lemmy-easy-deploy-lemmy-ui-1  |     at TLSSocket.socketErrorListener (node:_http_client:495:9)
lemmy-easy-deploy-lemmy-ui-1  |     at TLSSocket.emit (node:events:511:28)
lemmy-easy-deploy-lemmy-ui-1  |     at emitErrorNT (node:internal/streams/destroy:151:8)
lemmy-easy-deploy-lemmy-ui-1  |     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
lemmy-easy-deploy-lemmy-ui-1  |     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
lemmy-easy-deploy-lemmy-ui-1  |   type: 'system',
lemmy-easy-deploy-lemmy-ui-1  |   errno: 'ECONNREFUSED',
lemmy-easy-deploy-lemmy-ui-1  |   code: 'ECONNREFUSED'
lemmy-easy-deploy-lemmy-ui-1  | }

Any help would be appreciated (I did do a backup of the db volume btw)

Relevant: https://github.com/LemmyNet/lemmy-ui/issues/1530

Not sure if this is relevant: https://github.com/LemmyNet/lemmy/issues/3447

Diagnostic Information

==== Docker Information ====
Detected runtime: docker (Docker version 24.0.2, build cb74dfc)
Detected compose: docker compose (Docker Compose version v2.18.1)
   Runtime state: OK

==== System Information ====
KERNEL: 5.15.0-76-generic (x86_64)
SHELL: /bin/bash
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_CODENAME=jammy
UBUNTU_CODENAME=jammy
MEMORY:                total        used        free      shared  buff/cache   available
Mem:           7.8Gi       500Mi       3.0Gi        72Mi       4.3Gi       6.9Gi
Swap:             0B          0B          0B

==== Lemmy-Easy-Deploy Information ====
Version: 1.2.4

IMAGE                                     CREATED         STATUS
caddy:latest                              8 minutes ago   Up 8 minutes
ghcr.io/ubergeek77/lemmy-ui:0.18.1-rc.9   8 minutes ago   Up 8 minutes
ghcr.io/ubergeek77/lemmy:0.18.1-rc.9      8 minutes ago   Up 8 minutes
postgres:15-alpine                        8 minutes ago   Up 8 minutes
asonix/pictrs:0.4.0-beta.19               8 minutes ago   Up 8 minutes

Integrity:
    a787829f89e77dde4e4fbedd25c4fd94e44c483bb04735dc1b23e8b6d5b2fe37  ./deploy.sh
    1e9b0c0988998dcc33cb0fbfdb0e1679229424e724f898b797380adc7d102446  ./templates/Caddy-Dockerfile.template
    c1202e70662dd2228da36a35a0f38ec8fc81bec8964d7315d02e8671a58dd7d7  ./templates/Caddyfile.template
    2537678c7971df36c1ed95f4228d3cfcb15bb4a28a60d939eaf8dd75b5d64a36  ./templates/cloudflare.snip
    c494a610bcb4cd1cfc0a4fe4fb0f6d437b2a84a0ad1625daee240e6dd6f1c910  ./templates/compose-email-volumes.snip
    c9cb4c5fee12930e17798a02ae1bd12e2dc69e149a394c24511bc9d4e6b776d4  ./templates/compose-email.snip
    d9928baea61975bd432e1a4a7439c04468547c9b42e2bb483c6f18d9438665eb  ./templates/docker-compose.yml.template
    a74770730fd0e809d44e5247e1300be04c2b664be13d06bfec00296e3e63861b  ./templates/lemmy-email.snip
    c834cdce9eaf77f38155b404724fdfe66845575386ee516987452aa715642a6f  ./templates/lemmy.hjson.template

==== Settings ====
        CLOUDFLARE: No
 CADDY_DISABLE_TLS: false
   CADDY_HTTP_PORT: 80
  CADDY_HTTPS_PORT: 443
 LEMMY_TLS_ENABLED: true
      ENABLE_EMAIL: true
         SMTP_PORT: 587
    ENABLE_POSTFIX: false
POSTGRES_POOL_SIZE: 5

==== Generated Files ====
Deploy Version: 0.18.1-rc.9;0.18.1-rc.9

total 36K
drwxr-xr-x 2 0 0 4.0K Jul  3 02:58 caddy
-rw-r--r-- 1 0 0   33 Jul  4 08:07 caddy.env
-rw-r--r-- 1 0 0 1.7K Jul  4 08:07 docker-compose.yml
-rw-r--r-- 1 0 0   50 Jul  3 02:58 lemmy.env
-rw-r--r-- 1 0 0  730 Jul  4 08:07 lemmy.hjson
-rw-r--r-- 1 0 0   49 Jul  3 02:58 pictrs.env
-rw-r--r-- 1 0 0   37 Jul  4 08:07 postfix.env
-rw-r--r-- 1 0 0   51 Jul  3 02:58 postgres.env
-rw-r--r-- 1 0 0   24 Jul  4 08:08 version

[ubergeek77]

Yep. That's a Lemmy UI bug. It's caused by improper error handling in the server-side-rendering. If Lemmy-UI can't download the icon for the instance, it simply gives up, instead of proceeding without the icon like it should. You likely added an icon to your instance not too long ago.

Based on the discussion in that issue, it also seems like the URL it's trying to use is being incorrectly calculated in some cases. In your case, it's trying to use port 443 on a localhost IP. It should be using the pictrs internal hostname, but it isn't. So there's probably two bugs at the same time here.

Unfortunately, this is a Lemmy-UI bug, not a Lemmy-Easy-Deploy bug, so I can't help you fix it permanently. However, you can check that linked issue for some workarounds that seems to be working for people. You'll basically have to log in to your Postgres database from the command line and clear the icon manually. I've never done that, so I can't help you much more than that, but I can at least tell you how to get into your Postgres shell:

cd ./live
docker compose -p lemmy-easy-deploy exec postgres /bin/sh
psql -U lemmy

From there, you can probably run some of the postgres commands posted in that issue I linked.

I hope that helps you at least work around it for now!

[fmunim]

Yep. That's a Lemmy UI bug. It's caused by improper error handling in the server-side-rendering. If Lemmy-UI can't download the icon for the instance, it simply gives up, instead of proceeding without the icon like it should. You likely added an icon to your instance not too long ago.

Based on the discussion in that issue, it also seems like the URL it's trying to use is being incorrectly calculated in some cases. In your case, it's trying to use port 443 on a localhost IP. It should be using the pictrs internal hostname, but it isn't. So there's probably two bugs at the same time here.

Unfortunately, this is a Lemmy-UI bug, not a Lemmy-Easy-Deploy bug, so I can't help you fix it permanently. However, you can check that linked issue for some workarounds that seems to be working for people. You'll basically have to log in to your Postgres database from the command line and clear the icon manually. I've never done that, so I can't help you much more than that, but I can at least tell you how to get into your Postgres shell:

cd ./live
docker compose -p lemmy-easy-deploy exec postgres /bin/sh
psql -U lemmy

From there, you can probably run some of the postgres commands posted in that issue I linked.

I hope that helps you at least work around it for now!

@ubergeek77 Thank you for the response, could you atleast tell me what did you do to fix it? Since it does not seem like you went into the db?

[ubergeek77]

I just got lucky. When I had this issue, it was from before I updated the docker-compose.yml.template to remove the internal Docker networks. In Lemmy 0.17.4 and below, the Lemmy maintainers used to have each service on a special internal network that didn't have internet access. In 0.18.0, they removed those so that every service has internet access.

Before I made this change, Lemmy-UI was trying to reach the internet to go to my real domain to download the icon, but couldn't, so it would crash. I "fixed" it by updating docker-compose.yml.template, however:

• The UI didn't need internet access before 0.18.0, but now it does. This is a regression.
• Pictrs being unavailable shouldn't cause the entire frontend to error like this.

In your case (and some other people in that thread), it seems like Lemmy-UI is somehow resolving an IP address of 127.0.1.1 for your domain. That's obviously wrong, and you're not the only person with that issue. I have no idea what causes it though. This is sadly a regression in 0.18.0 that the Lemmy team will have to eventually fix. But since you are affected, removing the icon is the best you can do for now, sadly.

[fmunim]

@ubergeek77 Ah, thank you for the explanation. It seems to work if you remove the icon from DB. Thank you again!

[ubergeek77]

@granddao Out of curiosity, if you run cat /etc/hostname on your server (not in the postgres container), does it print out your exact Lemmy hostname? Like the .com and all?

[fmunim]

@ubergeek77 yes. It does. It seems like I can just upload back the icon after removing it but if anytime it restarts it will go back to that error.

[ubergeek77]

Ok, I see what's going on here. I'll have to add a warning about this.

Check out what happens if you do this:

cat /etc/hosts

You probably see something like 127.0.1.1 lemmy.your-domain.com, right?

If your machine hostname matches the domain you actually want to use for Lemmy, the record in /etc/hosts will override any DNS requests, and make the container think it needs to talk to 127.0.1.1, instead of the real IP.

I'm not sure what your hosting provider is, but you probably named your VM/VPS that domain name, causing it to become this hostname.

You will need to change the hostname of your server to avoid issues like this. You can look up how to do this for Ubuntu, but maybe this will be all you need:

https://phoenixnap.com/kb/ubuntu-20-04-change-hostname

Once you do that and reboot, check /etc/hostname and /etc/hosts to make sure your Lemmy domain isn't in there. Then, you can set an icon and you will be fine!

Still, Lemmy-UI needs some error handling :p

[fmunim]

@ubergeek77 wow you are a genius, that actually worked!!! Thank you ❤️❤️

I agree about that. Imagine an icon bringing the whole site crashing down.

[ubergeek77]

You're welcome!