The Road to 0.18.3 - upgrade items

[pallebone]

Hi again,

There were some security fixes that had to be applied so I have made a few changes to my docker-compose. These will need to be accommodated in your script for things to go well for 0.18.3 if possible... Hopefully this is something you will be able to implement.

Just to catch you up the docker compose file had to eventually be edited to ghcr.io/ubergeek77/lemmy:0.18.2 and ghcr.io/ubergeek77/lemmy-ui:0.18.2 just to get it to be right up to date this morning. These changes should be benign, however some other items would need accommodation:

1) asonix/pictrs:0.4.0 - had to set this tag so that object storage was supported by pictrs 2) These keys are required to be settable in your script:

  - PICTRS__STORE__TYPE=object_storage
  - PICTRS__STORE__ENDPOINT=https://<endpoint>.com
  - PICTRS__STORE__BUCKET_NAME=InsertNameHere
  - PICTRS__STORE__REGION=InsertRegionHere
  - PICTRS__STORE__USE_PATH_STYLE=false
  - PICTRS__STORE__ACCESS_KEY=KeyHere
  - PICTRS__STORE__SECRET_KEY=SecretHere

This is so the docker compose file has this underneath - PICTRSMEDIAGIF__MAX_FRAME_COUNT=400 in your resulting file.

3) Still keen to work on the backup with you. I am up to date with the staging branch :)

4) Possible option to use nginx instead of caddy? Can post a nginx config if you would like.

Kind regard Pete

[ubergeek77]

Just to catch you up the docker compose file had to eventually be edited to ghcr.io/ubergeek77/lemmy:0.18.2 and ghcr.io/ubergeek77/lemmy-ui:0.18.2 just to get it to be right up to date this morning.

You should not be modifying the docker-compose file manually. I print several warnings about this in the script (inspired by discussions with you), and I have mentioned this multiple times. To update the Lemmy stack to any version, run deploy.sh again.

asonix/pictrs:0.4.0 - had to set this tag so that object storage was supported by pictrs

My deployment uses the official deployment as a reference. Upstream Lemmy is still on beta 19, so mine is as well.

These keys are required to be settable in your script:

Please use customPictrs.env for these values.

Still keen to work on the backup with you. I am up to date with the staging branch :)

Thanks, I will work on it when I can 👍

Possible option to use nginx instead of caddy? Can post a nginx config if you would like.

Certainly not, the reason I use Caddy is because so many people had issues with Nginx. Caddy handles a ton of things automatically, including but not limited to, automatically enabling websockets, and automatic certificate generation and rotation.

[pallebone]

Problem is that there is a delay on your updating of the script and sometimes things move quickly as the developers are fixing things.

The pictrs beta you use is based on outdated documentation, nobody else is using it and when I joined the pictrs chat and asked the dev he said I needed to move to 0.4.0, as well so its not applicable to use an older version unfortunately.

I can include the custom file but still need to be on that later version.

Also regards caddy, nobody else is using it so having issue where they apply changes or fixes or improvements then i cant do similar things. Unclear if this will be an eventual major issue. Also some of them spoke/have used this or told me about this option https://hub.docker.com/r/jonasal/nginx-certbot that has certs and nginx together.

Just hitting a lot of inflexibility issues that make using the script purely and not making changes other ways difficult especially when critical changes have to be made and time is not available to wait etc.

Not sure your thoughts on this?

Kind regards Peter

[ubergeek77]

Problem is that there is a delay on your updating of the script and sometimes things move quickly as the developers are fixing things.

Please understand, I am not a Lemmy maintainer, and I do not claim to have the capacity to update things as fast as they can. I am trying my best, but I spend a significant amount of my spare time supporting issues here on GitHub, in addition to having a day job and other responsibilities. I only have significant time to work on this script on the weekends.

With that being said, I am available to push critical and time sensitive updates to this script, and I have monitoring and alerts set up to automatically notify me on any new Lemmy tags, and any changes to their reference files. There have not been any this week.

The pictrs beta you use is based on outdated documentation, nobody else is using it and when I joined the pictrs chat and asked the dev he said I needed to move to 0.4.0, as well so its not applicable to use an older version unfortunately.

I am not referring to the documentation. If there is an expectation that people should be on Pictrs 0.4, then they need to update the reference compose file, or otherwise tell me where to look where the "expected" version is communicated. The reference file is still on 0.4 beta 19:

https://github.com/LemmyNet/lemmy/blob/69a7181a29ea01672c61e0190e68b632dfcaf232/docker/docker-compose.yml#L75

Also regards caddy, nobody else is using it so having issue where they apply changes or fixes or improvements then i cant do similar things. Unclear if this will be an eventual major issue.

Using Caddy was an architectural choice for this project and I will not be moving to Nginx. Being able to deploy Caddy along with Lemmy was a very large part of my motivation behind this project. In addition, their Nginx config routinely has problems that the Caddy config does not have. Such as:

https://www.github.com/LemmyNet/lemmy-ansible/issues/106

Just hitting a lot of inflexibility issues that make using the script purely and not making changes other ways difficult especially when critical changes have to be made and time is not available to wait etc.

With respect, you seem to have certain expectations of this project that I have not committed to. I have committed to providing an easy out-of-the-box deployment for Lemmy that is simple for beginners to use. I do not expect beginners to want, or even know how to use, all these advanced options that appear to be an expectation of yours. And even in the case that those are desired, I provide advanced configuration options that you are welcome to learn how to use. I have significantly increased the original scope of this project due to your feedback, but unfortunately I think I need to draw a line somewhere.

If you want more parity with the "official" configuration, such as using Nginx, then perhaps this deployment is not for you.

[pallebone]

Sorry I am not trying to upset you, I am just providing my feedback and issues I am running into, and asking your opinion. Im not demanding anything of you or telling you what to do. Hope that is ok. Sorry if this was not communicated clearly or respectfully. It was not my intention to cause you any stress :(

[ubergeek77]

No hard feelings! I am not in their Matrix chat, so this feedback is illuminating and welcome. Though, as a single maintainer, I do have to make some concessions on what will be supported vs what will not. My intent was that the custom folder would be a nice middle ground.

As for the Pictrs version discrepancy, many people use the docker-compose file in the official repo, so I used that as my reference. Apparently, they are not as timely with updates to that, so I will start monitoring their Ansible docker-compose file instead.

As for the new Pictrs variables, using object storage is certainly an advanced use case, so I will continue to recommend that those variables are set in customPictrs.env. Custom environment variables are pretty much harmless, and will basically never break on updates.

And as for Nginx, no, sorry, I really cannot stand Nginx, the Caddy configuration is literally 10x smaller and much easier to automate 😅

[pallebone]

Regards pictrs reason they all use 0.4.0 is the devs of lemmy are using ansible and so they update ansible documentation more frequently. Here you can see they use 0.4.0 even though docker link you sent was not updated. i think its just an oversight because they simply use something else and have too much to do.

https://github.com/LemmyNet/lemmy-ansible/blob/main/templates/docker-compose.yml

" pictrs: image: asonix/pictrs:0.4.0

this needs to match the pictrs url in lemmy.hjson"

Again let me reiterate my apology as well as my confirmation I am not wanting to stress you out in any way. If I have offended you I am deeply sorry and would like to apologize for that.

If you are upset and would prefer not to communicate further with me I can understand and respect your decision, however if there is any way we can continue to communicate please let me know how I should moderate my behavior. Thank you in advance.

Kind regards P

[ubergeek77]

Everything is quite alright. Thank you for pointing out the Pictrs thing, I had no idea

[pallebone]

Apologies I see you have replied as I clicked send. Sorry for the out of order message. I see your reply and am accepting your decision.

[pallebone]

Hi Again,

Sorry to trouble you, just wondered if I could update you on some things?

Point 1) Assume that next release will use 0.4.0 pict.rs at minimum?

2) I have created a file in /lemmy/Lemmy-Easy-Deploy/custom called customPictrs.env I was unsure how to format the file. I has no spaces in front of the settings as such it is just a file with the below:

• PICTRSMEDIAFORMAT=webp
• PICTRSMEDIAMAX_FILE_SIZE=1
• PICTRSSTORETYPE=object_storage
• PICTRSSTOREENDPOINT=https://REDACTED
• PICTRSSTOREBUCKET_NAME=myservpic
• PICTRSSTOREREGION=Virginia
• PICTRSSTOREUSE_PATH_STYLE=false
• PICTRSSTOREACCESS_KEY=REDACTED
• PICTRSSTORESECRET_KEY=REDACTED

Is this correct format and will work?

4) I never want to ever speak about nginx ever again. CADDY WILL WIN!!!!!

——————-

3) For backup if there is any progress I can do some testing if you tell me to.

Kind regards Pete

[ubergeek77]

1. Yep, that's right, the docker-compose.yml that is generated will use 0.4.0 now
2. In the Advanced Configuration section, I do have a link to the Docker documentation about the specifics of that file, but that's basically it. Just VARIABLE=VALUE pairs on each line with no spaces around the =
3. I haven't done any more work on it, but the core backup functionality won't really change. Just some UX improvements i.e. backing up everything at once. I'll add something to dump the postgres db as a .sql too, but it will basically be the same as running it for yourself
4. Caddy is quite nice yes :p

[pallebone]

Thank you,

For the point 2, custom pictrs file - should it have a leading '-'

ie is it this in the file on each line: "- PICTRSMEDIAFORMAT=webp" or this: "PICTRSMEDIAFORMAT=webp"

Please ignore ", this is just to show what exactly I have typed.

Kind regards Peter

[ubergeek77]

In the advanced configuration page, I have a link to the Docker documentation that provides information on the correct syntax:

These files follow the Docker environment file syntax (basically just VAR=VAL).

That link directly is:

https://docs.docker.com/compose/environment-variables/env-file/

But in short, no, just VAR=VAL on separate lines and nothing else.

[pallebone]

Ok also one question about the backup scrip, this line here:

docker run --rm -v lemmy-easy-deploy_${FROM_LED_VOLUME:?}:/from:ro alpine ash -c 'tar -czvf - -C /from .' >${TO_TAR_GZ:?}

This has this in the export to tar.gz : 'tar -czvf - -C /from

why is there -czvf then a space then - then another space then -C?

Is this correct? Or can it be: 'tar -czvf -C /from

unclear what the dash with no switches does to me.

Kind regards Peter

[ubergeek77]

Using - as an argument to tar tells it to write to stdout. This lets me stream data to the output file with > without having to deal with mounting the output folder into Docker.

[pallebone]

Ok thank you. I think we are ready for 0.18.3 then if you are not adding anything further to the script before then. Appears to work to tar.gz file as I wanted.