Docker-free installation process

[derneuere]

I got the request for a docker-free installation process. Jellyfin for Example has docker images but also has binaries for windows, macOS and Linux.

I think we have to use pyinstaller for that. We have to bundle portable versions of the dependencies like PostgreSQL, Redis, Nginx and Node.js with it.

To build all the binaries we should probably have a CI/CD process in place.

We would also need to create an optional settings file where you can configure the parameters we currently set via docker-compose

[strugee]

We have to bundle portable versions of the dependencies like PostgreSQL, Redis, Nginx and Node.js with it.

No no, don't do this. Or at least make it optional. Instead just let users point the code at their own Postgres, Redis, etc. and document how to configure the web server. I'm also interested in a Docker-free install, and while a big part of that is that I don't like Docker's security problems, another part of it is that operationally I want to be running one Postgres database, not a different Postgres database for every app that uses Postgres.

[ekianjo]

also interested in a docker free install.

[atomicwrites]

@strugee I haven't tried it but the docker-compose file has variables that you can change for the Postgres host, port, user, password, and database name. I don't think anything is stopping you from using an existing shared postgres in a container or not in docker at all. You just need network connectivity.

[strugee]

Yeah, I saw that too. I'm manually deployed several Python apps before so I think I should be able to figure it out, but I need some hints as to what exactly needs to be deployed and where the code for those components is. There's the backend, the proxy, and the frontend right?

If someone gives me a quick overview of how you'd do it I can figure out the details and write up some install instructions.

(To clarify: I also don't like Docker for other reasons. Primarily the fact that many images are chock-full of security vulnerabilities and there's no good/non-hacky way to deal with that unless you're a company and have an entire team dedicated to keeping images up-to-date.)

[derneuere]

@strugee You can find the frontend , backend and proxy here. You can find the relevant backend settings here, I think the relevant file for the frontend are here, here, but I am not sure about the last one. Never worked on the proxy, if you have any concrete questions I can route them to @sim4city. He worked on it for a couple of commits.

[foobazbar123]

@strugee You can find the frontend , backend and proxy here. You can find the relevant backend settings here, I think the relevant file for the frontend are here, here, but I am not sure about the last one. Never worked on the proxy, if you have any concrete questions I can route them to @sim4city. He worked on it for a couple of commits.

I'd agree that not building things shouldn't be done for a Docker deployment. That's the Docker way, after all :) There's no need to do so for VM or LXD style deployment and it would just create more overhead for you guys, for little real benefit.

I'm going through some of the files you linked to in your post, specifically the NGINX parts at the moment. I presume "backend" is a name that would resolve in the Docker environment to a particular container and "frontend" as well?

Based on that, is it correct to say that the backend process then listens on port 80, expecting http traffic and the frontend process listens on port 3000 by default? Is it expected that external clients, web users, are to be passed into the backend on the api and media end points?

I ask this as it would be nice to break the current dependency on nginx so a person could use caddy or whatever if they wanted. Shouldn't be hard to come up with an example caddy config based on what I'm seeing at the moment :)

@strugee I just wrote all I said above and then really read your comment. Looks like "proxy" is just NGINX in this case and nothing fancy, which makes things easier. @derneuere's post contains the bits for the front end dependencies, so I'm going to try and get that deployed today into an LXD container.

[foobazbar123]

Well, I have it up and running. Only thing is, the initial user isn't created so I can't login and test anything. Is there any way to force the creation of the user? I've got the ADMIN_USERNAME and ADMIN_PASSWORD environment variables set, but nothing is created. Thoughts?

[derneuere]

The admin user is created in the entrypoint script: https://github.com/LibrePhotos/librephotos/blob/dev/entrypoint.sh You could change it to force the creation of an admin acount.

[foobazbar123]

Ah, I see what I did wrong. I figured the migrations were doing that and totally over looked the bottom part there with manage.py shell part :)

[foobazbar123]

Okay, adding in the bits to create the account has it working properly. I'm going to feed it some photos and see if everything appears to be correct. If it does, I'll dump what I've done in another comment

[foobazbar123]

It's running, mostly. There are various issues resulting from, I believe, the "proxy" usage of NGINX. Is there expected to be multiple nginx things running in various directions? There's an nginx conf for the frontend, backend and the "proxy". Am I correct in thinking that with the current design three nginx instances are required?

[derneuere]

AFAIK it is only one Nginx instance. The conf is here

I am pretty sure the other nginx.conf in the other repositories are not used. @sim4city do you know if the other nginx.conf are used?

[strugee]

@foobazbar123 just skimmed through your comments on my phone - I'll probably hop on a computer in an hour or two and try setting things up too. Let's chat? I just created the #librephotos IRC channel on Freenode if that works for you

[strugee]

Oh, I forgot there was a Discord server... we could do that too although honestly because Discord does not respect my privacy or freedom it would be great to avoid it. I wonder how difficult it is to bridge Discord and IRC - I'll file a separate GitHub issue for that so we don't clutter this one up.

[foobazbar123]

Matrix is better at bridging things than IRC/Discord :)

[foobazbar123]

I have it fully working, so far as I can tell, now. I've got some working.. er.. instructions, if they can be called such, that will take one from a more or less blank LXD instance (with a couple of PPAs for Postgres and Node) to "working system"

It appears to one degree or another at least two nginx instances are probably desired, and possibly all three, in the Docker config. Consolidating the settings into one is possible in the LXD environment so I just did that. Prior to doing so, thumbnails and such wouldn't work

[thequailman]

I am also interested in doing this. I've read over the Makefiles and it seems pretty simple to build, however it would help me out immensely if you'd consider adding:

1. Releases/tags consistent across the various component repos
2. Combine everything into one repo (simplify the build process)

For sure until 1 is done, I don't think it's feasible to do this--there is no way to figure out what commits match.

[derneuere]

@thequailman

Would be great if you want to tinker with it a bit 👍 If you want to build a binary you should look at the dockerfile. The makefile only creates a docker image.

1. The latest dev commits are always compatible with each other.
2. I don't think that we will merge the repos. Concerns should be separated to allow for greater modularity. It also should not be that hard to do it in a makefile. Git clone the repos in the makefile, and you will have all the code in one place.

[derneuere]

You can now install librephotos locally under debian11: https://github.com/LibrePhotos/librephotos-linux

[foobazbar123]

Running through the install instructions as written, on Ubuntu. I did find a couple of issues. The first being, the git clone command doesn't work, as written. It is wanting an SSH style clone, but the repository in question doesn't have that enabled. Changing either the command or the repository would sort that. Also, you are calling bash as /usr/bin/bash, but I'm not sure which distro, if any, would have bash there. I checked the file listing for Debian and it, along with Ubuntu and I think all major distros, have it at /bin/bash. I updated the install script on my end to use /bin/bash and it proceeded. On Ubuntu, nodejs and npm are mutually exclusive packages, as nodejs contains npm, or at least it is on my system. I do have the NodeSource repository on my LXD test container, so that may be a local issue on my part.

For the default configuration file, I'd suggest making the DB_HOST default to localhost. Currently, it gives something that appears to be an actual FQDN. It isn't, but it looks like one and that could confuse some people. :) Ditto for the Redis host. The instructions also say to set "Postgresql information", but not what that actually is. Not the values, but what you're referring to. I know what it is, but others who read it may not, you know? Nothing is mentioned about setting the timezone, though I imagine one would probably want/need to do so :)

/usr/lib/librephotos/bin/librephotos-createadmin <-- has the same /usr/bin/bash issue as the install script.

In the postgres information, you have it setting things up for a French user. That's fine and all, but could confuse the daylights out of someone who isn't used to such a thing and doesn't know how to change it. A note on what's going on there, and what to set it to for some common alternatives may be useful. And speaking of the DB, at this point I'm stuck. With the instructions, as written, one ends up with a blank DB (unless I missed something). As a result, you cannot create an admin user and I presume even start the app.

Sorry if this is a bit flow of thought, as it were. Just wanted to get the thoughts out there as I was going through the instructions :)

On Fri, Mar 12, 2021 at 6:44 AM Niaz @.***> wrote:

You can now install librephotos locally under debian11: https://github.com/LibrePhotos/librephotos-linux

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LibrePhotos/librephotos/issues/57#issuecomment-797467221, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIA3Q5XRDGX5BAXMXRCFNYTTDIEBTANCNFSM4VO4AB5A .

[atomicwrites]

@foobazbar123 Talking specifically about the /usr/bin/bash issue, /bin is just a symlink to /usr/bin on most modern distros. It's called the /usr merge. According to https://wiki.debian.org/UsrMerge it is the default on new installations but the change has to be applied manually to existing ones. It is weird to call the command in /usr directly though, the symlinks are there to preserve compatibility and not using it will break your script on some systems as you found out.

[tomamplius]

yes but maybe that the best way is use

/usr/bin/env bash

I will check compatibility with ubuntu 20.04 and take into account some input

[tomamplius]

@foobazbar123 thanks for you test. It was only test on debian 11 before. You can find a pull request to Ubuntu 20.04 https://github.com/LibrePhotos/librephotos-linux/pull/2 If you can test and validate the new package from https://github.com/tomamplius/librephotos-linux/archive/ubuntu20.04.zip

About nodejs and npm are exclusive. Can you give me more information. I can't reproduce

[strugee]

I do have the NodeSource repository on my LXD test container, so that may be a local issue on my part.

This is indeed a local issue. NodeSource ships npm inside their nodejs package, but the nodejs package in the Debian/Ubuntu archives separates it out into the npm package.

[tomamplius]

OK i understand.

[foobazbar123]

Yeah, I suspected it would turn out to be a local issue. No worries :) Sure, I can generate another LXD container and give it a bash :)

On Sat, Mar 13, 2021 at 8:06 AM AJ Jordan @.***> wrote:

I do have the NodeSource repository on my LXD test container, so that may be a local issue on my part.

This is indeed a local issue. NodeSource ships npm inside their nodejs package, but the nodejs package in the Debian/Ubuntu archives separates it out into the npm package.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LibrePhotos/librephotos/issues/57#issuecomment-798420288, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIA3Q5VRJMBNIB53ZKC2INLTDNPM7ANCNFSM4VO4AB5A .

[foobazbar123]

By the way, it looks like the git clone was fixed, but then the fix was reverted and it is back to using the SSH clone :)

On Sun, Mar 14, 2021 at 11:57 AM FooFoo Foo @.***> wrote:

Yeah, I suspected it would turn out to be a local issue. No worries :) Sure, I can generate another LXD container and give it a bash :)

On Sat, Mar 13, 2021 at 8:06 AM AJ Jordan @.***> wrote:

I do have the NodeSource repository on my LXD test container, so that may be a local issue on my part.

This is indeed a local issue. NodeSource ships npm inside their nodejs package, but the nodejs package in the Debian/Ubuntu archives separates it out into the npm package.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LibrePhotos/librephotos/issues/57#issuecomment-798420288, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIA3Q5VRJMBNIB53ZKC2INLTDNPM7ANCNFSM4VO4AB5A .

[tomamplius]

check the branch ubuntu20.04 I'ts not commit https://github.com/tomamplius/librephotos-linux/tree/ubuntu20.04

[foobazbar123]

Yeah, realized that right after I said it haha

On Sun, Mar 14, 2021 at 12:05 PM tomamplius @.***> wrote:

check the branch ubuntu20.04 I'ts not commit https://github.com/tomamplius/librephotos-linux/tree/ubuntu20.04

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LibrePhotos/librephotos/issues/57#issuecomment-798942177, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIA3Q5RXHSEUY6XMAMIDU33TDTUG7ANCNFSM4VO4AB5A .

[foobazbar123]

It's.. mostly working? I'm running into various issues which I admit I kinda lost track of. One thing I'd suggest is breaking out the "apt install" steps from the install script so that the install script can just be called as the librephotos user. The apt installs could be done either by cut and paste on the user's part or a separate script or something.

Part of why I lost track of any issues was I forgot to take notes. However, the other thing is the current script has zero error handling and one just kinda has to run it over and over until it works. More or less. I got through it and the app is started. However, as previously noted, the DB is blank. No schema, no tables, no nothing. I presume there is an import or other step currently missing somewhere? :)

On Sun, Mar 14, 2021 at 12:06 PM FooFoo Foo @.***> wrote:

Yeah, realized that right after I said it haha

On Sun, Mar 14, 2021 at 12:05 PM tomamplius @.***> wrote:

check the branch ubuntu20.04 I'ts not commit https://github.com/tomamplius/librephotos-linux/tree/ubuntu20.04

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LibrePhotos/librephotos/issues/57#issuecomment-798942177, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIA3Q5RXHSEUY6XMAMIDU33TDTUG7ANCNFSM4VO4AB5A .

[tomamplius]

added on readme /usr/lib/librephotos/bin/librephotos-upgrade

[foobazbar123]

Got it. That sorted the DB issue :) Poking around now

On Sun, Mar 14, 2021 at 12:51 PM tomamplius @.***> wrote:

added on readme /usr/lib/librephotos/bin/librephotos-upgrade

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LibrePhotos/librephotos/issues/57#issuecomment-798949181, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIA3Q5UYTDNDD63RAIMYIELTDTZTBANCNFSM4VO4AB5A .

[foobazbar123]

Seems to be working fine. Was able to import some photos from the filesystem and do the things one would expect with them.

So, in summary for suggestions:

1. Break out and document the system dependencies. Bonus points for putting them either in and cut and pastable commands or list.
2. If you do the first one, that allows you to remove them from the install script and simplify it. This also eliminates issues that may result like I had with the node source repository being present
3. I'd suggest not installing nginx by default, and instead suggesting it in the documentation with either an included suggested config or better yet, include it in the documentation. As to why do this, one might want to use something aside from NGINX, like Caddy. This also allows for further simplification of the install script. If you really want to install something by default, I'd toss a recommendation for Caddy. It's simpler and supports automatic TLS. Does have the downside of not necessarily working as simply for internal installs. So, upsides and downsides I suppose.
4. Far better error handling in the script, or eliminate the script, or drastically simplify it. It's a noble idea to make it as close to fully automatic as possible, but has the downside that if anything goes sideways, the user may have no idea what happened, what it was trying to do, or anything else that could tell them what went wrong. "It just failed :(" would be the result. Also, along these lines, add a message at the end saying whether it worked or failed. Right now, it just kinda ends and for some that's fine, but for J.Random User, they could be left wondering what happened

I think that's all that comes to mind at the moment :)

On Sun, Mar 14, 2021 at 12:53 PM FooFoo Foo @.***> wrote:

Got it. That sorted the DB issue :) Poking around now

On Sun, Mar 14, 2021 at 12:51 PM tomamplius @.***> wrote:

added on readme /usr/lib/librephotos/bin/librephotos-upgrade

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LibrePhotos/librephotos/issues/57#issuecomment-798949181, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIA3Q5UYTDNDD63RAIMYIELTDTZTBANCNFSM4VO4AB5A .

[tomamplius]

@foobazbar123 thank you for your test I will take suggestions in the next version @derneuere Please can you merge https://github.com/LibrePhotos/librephotos-linux/pull/2 after this test?

[derneuere]

@foobazbar123 Thanks for the test! @tomamplius I merged it :)

[UndarkAido]

I tried to set up Caddy and posted LibrePhotos/librephotos-linux#8 but I can't quite get it to work. Not really my skillset.

[thequailman]

What's the reasoning for not serving the UI from the backend? Why have a dependency on a proxy?

[tomamplius]

dependency become optional on the next version https://github.com/tomamplius/librephotos-linux/tree/v2

[thequailman]

Sorry, that's not quite what I'm looking for. Why can't the backend serve these files directly instead of having another daemon serve them?

[tomamplius]

I mean it's choice because webserver like nginx apache are more perform than django to server files.

[thequailman]

The way it currently works, the frontend is built everytime time it's restarted and served from node/express. Instead, it could be built once and served as static files from django.

NGINX/Apache don't serve these files directly, they proxy to the node server. You could setup caching for either situation to have them serve the files from cache, but I highly doubt there's any kind of performance impact serving files from django for 99% of librephoto setups.

[tomamplius]

Location /protected_media /original /nextcloud_original are not used? i have not currently check this part. https://github.com/tomamplius/librephotos-linux/blob/v2/ressources/nginx/nginx.conf

[tomamplius]

and /media /api are proxy to the backend not ton node server

[tomamplius]

maybe we can remove node server, right

[thequailman]

Disregard, I incorrectly assumed django can share files.

I created an Ansible task that does most of your script (you'll need to manually setup the nginx bit): https://gist.github.com/thequailman/caffade8049ccbee64113c77f42b9bbe#file-librephotos-md

I could probably clean this up and make it into a real ansible collection if anyone wanted that.

[tomamplius]

I don't know / use ansible. but it's a idea to add to add your script on official repo. @derneuere can you i add the script in librephotos-linux? or you want create librephoto-ansible?

@thequailman tkx and great work.

[ma-gh]

I am on the noober side of things, but I tried to do the linux install.

It crashed when nginx tried to bind where Apache lived. I wish I knew that this auto install was going to throw nginx in there. It isn't mentioned in the README. I hope I didn't bork my system but I just uninstalled nginx and deleted all the librephotos files.

I will return on a later release. Excited to try it out as a Google Photos replacement.

Thanks

[audioscavenger]

i am on the noober side of things, but i would gracefully appreciate that we concentrate on bugfix rather than starting new projects lol