Closed pallebone closed 1 year ago
ubergeek77
commented
1 year ago Not much I'll be able to do here sadly. You can check the generated docker-compose.yml to make sure it's being correctly mounted into the container, but beyond that I'm not knowledgeable of Postgres configuration specifics.
It could also be that this file is only considered when the database is created. I'm not sure.
ubergeek77
commented
1 year ago Ugh. This is so poorly documented. I'm not using the right path, it needs to be
/var/lib/postgresql/data/postgresql.confI can fix this later tonight, or you can temporarily modify your generated docker-compose.yml
pallebone
commented
1 year ago My friend, you know I would NEVER modify my docker-compose file while using lemmy easy deploy :)
Its ok thought its not urgent so I can wait and then test when its fixed and tell you if it resolved the issue.
Pete
ubergeek77
commented
1 year ago @pallebone This ought to do the trick:
https://github.com/ubergeek77/Lemmy-Easy-Deploy/releases/tag/1.2.9
pallebone
commented
1 year ago Sadly I am having a bit of a nightmare. After doing a git pull and ./deploy.sh -f everything broke with a failure. I have attached the failure log below.
I tried restoring the volumes of the containers first but this did not resolve the issue so I reverted that change and instead restored a copy of my lemmy/Lemmy-Easy-Deploy directory from backup and this resolved the issue and would start lemmy again.
Logs are below that I see an issue:
root@lemmy01:/lemmy/Lemmy-Easy-Deploy# cat failure-1691328044.log
lemmy-easy-deploy-postgres-1 |
lemmy-easy-deploy-postgres-1 |
lemmy-easy-deploy-postgres-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
lemmy-easy-deploy-postgres-1 |
lemmy-easy-deploy-postgres-1 |
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9119034,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
lemmy-easy-deploy-proxy-1 | {"level":"warn","ts":1691328035.9162407,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddy
file","line":2}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9179797,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhos
t:2019","//[::1]:2019"]}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9185028,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS",
"server_name":"srv0","https_port":443}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9186175,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003eed00"}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9186728,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9199178,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.920066,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9202857,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/q
uic-go/wiki/UDP-Buffer-Sizes for details."}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9206133,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9207954,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9208112,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["lemmy.myserv.one"]}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9210422,"logger":"tls","msg":"finished cleaning storage units"}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9221904,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
lemmy-easy-deploy-proxy-1 | {"level":"info","ts":1691328035.9223855,"msg":"serving initial configuration"} cd /leNote: I have not made any modifications to caddy AFAIK
pallebone
commented
1 year ago caddyfile if it helps:
root@lemmy01:/lemmy/Lemmy-Easy-Deploy/live/caddy# cat Caddyfile
(caddy-common) {
encode gzip
header {
-Server
Strict-Transport-Security "max-age=31536000; include-subdomains;"
X-XSS-Protection "1; mode=block"
X-Frame-Options "DENY"
X-Content-Type-Options nosniff
Referrer-Policy no-referrer-when-downgrade
X-Robots-Tag "none"
}
}
{$LEMMY_HOSTNAME} {
import caddy-common
reverse_proxy http://lemmy-ui:1234
@lemmy {
path /api/*
path /pictrs/*
path /feeds/*
path /nodeinfo/*
path /.well-known/*
}
@lemmy-hdr {
header Accept application/*
}
handle @lemmy {
reverse_proxy http://lemmy:8536
}
handle @lemmy-hdr {
reverse_proxy http://lemmy:8536
}
@lemmy-post {
method POST
}
handle @lemmy-post {
reverse_proxy http://lemmy:8536
}
}
root@lemmy01:/lemmy/Lemmy-Easy-Deploy/live/caddy#
I think I may have found a difference, after checking the moved directory that ran ./deploy -f I see this is how it modified the caddyfile:
root@lemmy01:/lemmyold/Lemmy-Easy-Deploy/live/caddy# cat Caddyfile
(cloudflare_https) {
tls {
dns cloudflare {env.CF_API_TOKEN}
}
}
(caddy-common) {
encode gzip
header {
-Server
Strict-Transport-Security "max-age=31536000; include-subdomains;"
X-XSS-Protection "1; mode=block"
X-Frame-Options "DENY"
X-Content-Type-Options nosniff
Referrer-Policy no-referrer-when-downgrade
X-Robots-Tag "none"
}
}
{$LEMMY_HOSTNAME} {
import caddy-common
import cloudflare_https
reverse_proxy http://lemmy-ui:1234
@lemmy {
path /api/*
path /pictrs/*
path /feeds/*
path /nodeinfo/*
path /.well-known/*
}
@lemmy-hdr {
header Accept application/*
}
handle @lemmy {
reverse_proxy http://lemmy:8536
}
handle @lemmy-hdr {
reverse_proxy http://lemmy:8536
}
@lemmy-post {
method POST
}
handle @lemmy-post {
reverse_proxy http://lemmy:8536
}
}
root@lemmy01:/lemmyold/Lemmy-Easy-Deploy/live/caddy#Some logic there is broken?
ubergeek77
commented
1 year ago Which service is failing? What do you see during deployment before the failure log is dumped?
I suspect it's Postgres, and it may not like the custom options you've passed in.
pallebone
commented
1 year ago Well because Caddy does not run correctly, lemmyui can not connect to postgres either. However I believe Caddy is the root cause because the error for postgres is just a generic, cant make a connection to postgres on the port error. I would guess this is just because caddy is not proxying the connection at this point.
ubergeek77
commented
1 year ago I don't see anything wrong with those logs or the generated Caddyfile.
What happens if you remove the customPostgres.conf file?
pallebone
commented
1 year ago Ok will do some troubleshooting tonight. From what I posted above I really think its the caddyfile though. Im thinking it does not like the cloudflare part.
Anyway will be easy to remove the postgres file and check if still broken and replace the caddyfile with a good copy. Will do it later when Im home.
P
ubergeek77
commented
1 year ago Nothing really jumps out at me about that Caddy config 🤔
1.2.9 was a one line change that didn't touch anything Caddy related, I only corrected the path used for custom Postgres configs to fix this issue.
That failure log is probably missing the full context. If Caddy is really failing, there would be at least one clear error message. And when deploying, you should have clearly seen Checking proxy... FAILED, or something, pointing out Caddy as the culprit.
Another possibility is that your Cloudflare token either got rotated or expired.
pallebone
commented
1 year ago Interesting. I thought this line was relevant but will try some stuff no problem later.
lemmy-easy-deploy-proxy-1 | {"level":"warn","ts":1691328035.9162407,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddy file","line":2}
ubergeek77
commented
1 year ago Yeah it always complains about that, but it's never an issue and definitely not fatal. One space out of place and it will nag.
pallebone
commented
1 year ago Ok so you are obviously a genius and are correct the failure was with postgres and the customPostgressql.conf file but the weird thing is if just an empty file exists it stops lemmy from working and postgres cannot start anymore.
So I tested with removing the file in the custom directory and everything starts fine, but then I did a 'touch customPostgressql.conf' file and then running a deploy.sh -f means it cannot start. I had tried with valid values generated from https://pgtune.leopard.in.ua/ prior to a blank file so having either good values or a blank file results in breaking lemmy when deployed.
Why do you think that would happen?
Kind regards P
pallebone
commented
1 year ago I figured out what is happening, I had believed this file was adding my customization to the deployment, as it suggests in the lemmy documentation. However it is actually overwriting the file and removing parts of it that are needed by default.
So the default values that need to be put back are:
listen_addresses = '*' dynamic_shared_memory_type = posix log_timezone = 'UTC' datestyle = 'iso, mdy' timezone = 'UTC' lc_messages = 'en_US.utf8' # locale for system error message lc_monetary = 'en_US.utf8' # locale for monetary formatting lc_numeric = 'en_US.utf8' # locale for number formatting lc_time = 'en_US.utf8' # locale for time formatting default_text_search_config = 'pg_catalog.english'
Then you can add your custom options afterwards such as max_connections and so on generated by the website I linked.
I have added all required to the file and its now working. This was not clear I had to do this. Sorry my bad :).
Pete
ubergeek77
commented
1 year ago Ah, and here I had a response written and ready to go!
Well, at least I can post this much:
If you specify an empty file, you end up providing postgres with a config file that's blank. If it didn't exist at all, postgres would generate its own, but since it exists but is blank, Postgres is unable to parse it, thus fails. Think of it this way - Postgres can't tell the difference between a blank config file, and a config file that has syntax errors. If it fails to parse, it just crashes.
Similar situation - those options were needed, hence it crashed. Good to know!
You'd think the Lemmy instructions on using the config generator they point to would have some kind of record of this information, some kind of knowledgebase, documentation, if you will...
pallebone
commented
1 year ago Its weird because I formatted my initial file like they suggested here: https://github.com/LemmyNet/lemmy-ansible/blob/main/examples/customPostgresql.conf
Also there is an error on that file also. If you specify temp_file_size=
Sorry again, and thanks again :)
Pete
pallebone
commented
1 year ago Changes appear to have helped (changes just before midnight) so memory seems better - hopefully will continue long term:
Did you check the FAQ & Troubleshooting section for answers to common questions and issues?
Describe the issue
Not respecting customPostgresql.conf
Hi there,
I have noticed that the file in /custom/custom/Postgresql.conf is not respected by Lemmy Easy Deploy.
I noticed because I installed pgAdmin4 and connected to the database. The values set on the database are the defaults and not what I configured in the customPostgresql.conf file.
How can I troubleshoot this issue?
Kind regards Peterostgresql.conf file.
How can i troubleshoot this issue?
Kind regards Peter