Feature Request: Docker installation

[renatobellotti]

I would love to give self-hosted etesync a try, but I'd like to run it on docker. I'm willing to try this on my own, but I will probably need some time to get it right... I will share my results once I have a working solution!

[daftaupe]

Hello @renatobellotti, maybe you want to check the image mentioned at https://github.com/etesync/server/issues/12#issuecomment-625870434. I don't use Docker myself though.

[victor-rds]

I would love to give self-hosted etesync a try, but I'd like to run it on docker. I'm willing to try this on my own, but I will probably need some time to get it right... I will share my results once I have a working solution!

Please try my image at https://hub.docker.com/r/victorrds/etesync, and tell me if it work for you, and post any issues on my repo

[BeatLink]

Awesome! Any idea when we can see 2.0 builds?

[worldofgeese]

@victor-rds do you intend to update your images to 2.0?

[victor-rds]

@worldofgeese Yes, just hadn't the time for doing it. The new Etesync 2.0 or Etebase has database breaking changes, @tasn and I talked a bit about what is the best course, just updating my images :latest tag can break it to almost everyone who uses it, so I got 2 options go and break or create a new separate image and add a deprecation notice.

[tasn]

As I mentioned it, I really dislike creating a new image as it'll lose all of the "reputation" (stars/downloads), which means people won't know it's the preferred choice. How about adding a tag legacy and then just update latest? This way, if people update without looking they'll have an easy way to revert back to a working version, while not losing all of the reputation this repo already got.

[victor-rds]

It's just a renaming because the repository already has versioned tags like v0.3.0

[tasn]

Ah right, there are already tags! So I see no problem with just having latest point to 2.0...

[victor-rds]

@tasn does the etesync/etesync-web project contains the migration tool? Or is just the client for 2.0?

[tasn]

Only the old etesync-web (so the legacy branch), so not the client for 2.0. https://client.etesync.com/migrate-v2/ Or you can migrate directly from the Android app.

[victor-rds]

@tast Is there an easy way to test if the database is from Etesync 1.0? I'm changing the entrypoint script for etebase, and I want to avoid damaging the original database. o support both databases (sqlite and postgres) I'm using python manage.py inspectdb journal_journal to find if the database is from etesync and not etebase..

[tasn]

Your check sounds to me as good as in. I don't have a better idea.

[victor-rds]

I don't know if this worth open a new issue, but the new etebase uses a ini config file, the location of this file is defined on the etebase_server/settings.py, could we set the etebase-server.ini via environment variable?

Here is the rationale: the docker swarm offers a secrets feature, mounting sensitive files to /run/secrets/<secret_name>, and the ini files can contain sensitive data, like database passwords. We already use the secret.txt in this way, also other orchestration tools like kubernetes has it's on secrets vaults, using and env var solve for all of them.

[tasn]

Sure thing, got an environment variable name in mind? Happy to do it. What I do locally is slightly different: I just symlink the path to the secrets path in my Dockerfile. So I'd have ln -s /run/secrets/etebase_ini /app/etebase-server.ini. Though I guess it's much easier to not hardcode it and just use an env var. :P It's not an option for my case (because I do it for the etebase_local_settings.py file which needs to have a specific location anyway).

[victor-rds]

Sure this works, but I'm trying to keep it generic for many use cases as possible. Another advantage, for me at least, is migrating all settings and db files to a single directory /data, and at same time avoiding touching the base dir with a symlink.

Maybe ETEBASE_INI_PATH to maintain consistency with ETEBASE_DB_PATH

config_locations = [os.environ.get('ETEBASE_INI_PATH', os.path.join(BASE_DIR, 'etebase-server.ini')), '/etc/etebase-server/etebase-server.ini']

[victor-rds]

I can make a pull request if you prefer

[tasn]

I replied to the PR! Looks great overall, just made two small comments, and after that I can merge it in. Thanks!

[tasn]

Btw, could you please tag the latest legacy release with the legacy tag (maybe also add a mention in the README), just so people know what's up.

[victor-rds]

Btw, could you please tag the latest legacy release with the legacy tag (maybe also add a mention in the README), just so people know what's up.

Already done ten days ago, I just didn't update the README: victorrds/etesync:legacy

[tasn]

Ahh, great job! :)

[BeatLink]

Probably a bit too late to chime in on this, but wouldn't it be far simpler to create a completely new docker image for EteBase? That way you don't need to worry about complex configurations, migrations and clashes, you could focus on making them clean and separate.

Hell, all you'd need to do is clone the existing docker image to a new repo, rename everything to EteBase, add any needed changes to the docker image, point to the latest EteBase version with the latest tag and publish to Docker.

Then the end user could download the separate docker image, spin up the new EteBase docker container alongside the legacy EteSync, fire up one of the migration tools with the 2 separate servers, migrate their data then shut down the old EteSync docker container. Easier for you guys and the end users. You guys can keep things clean and simple without worrying about compatibility or checking which version of the database is there. End users can keep using the legacy images as long as they need and nothing will break, and needing to spin up a separate docker container for EteBase means they have to take deliberate steps to migrate to the new architecture which conveys the backwards incompatible nature of this new release.

[tasn]

@BeatLink, we already considered that, though I think the current approach is more favourable for two reasons:

1. There are already 100k+ downloads and a few stars on docker hub. This there implies this is the correct and trusted image to use, and lends a lot of credibility to both the docker image in particular and the project in general. Losing this would be unfortunate.
2. There are sometimes incompatible software versions (when major versions change), and this one is no different. It's the same way you have Python2 and Python3 docker images. And Postgres 12 and 13. This case is not different.

[BeatLink]

Hmm, I understand. In any case, I'm working on a separate image right now. I've invited you to the Github organization and the Dockerhub organization as well.

[worldofgeese]

How does one migrate databases between 1.x and 2 given they simply flip the tag to latest or are already running latest?

On November 7, 2020 5:02:23 PM GMT+01:00, Tom Hacohen notifications@github.com wrote:

@BeatLink, we already considered that, though I think the current approach is more favourable for two reasons:

1. There are already 100k+ downloads and a few stars on docker hub. This there implies this is the correct and trusted image to use, and lends a lot of credibility to both the docker image in particular and the project in general. Losing this would be unfortunate.
2. There are sometimes incompatible software versions (when major versions change), and this one is no different. It's the same way you have Python2 and Python3 docker images. And Postgres 12 and 13. This case is not different.

[BeatLink]

https://github.com/etesync/server/blob/master/README.md#updating-from-version-050-or-before

You would have to use a migration tool. Since you cant use the same database its likely you would have to spin up a new docker container as well. This is why I think it's best to start with separate docker images. Because its entirely incompatible. Combining the two into a single docker image with only tags to separate them is just begging for builds to break, especially when several self hosters have scripts to automatically update their images to the latest build, which wouldn't take into account the breaking changes.

[BeatLink]

Even the above mentioned link states you have to use seperate install paths since the databases are incompatible

[tasn]

@BeatLink, as I said though, it's exactly like Postgres. You need to stop the server, migrate to a new one, and then run the new image.

[BeatLink]

Okay, I see. So I'll just go ahead and build an omnibus docker image for my personal use

[tasn]

What do you mean by omnibus?

Anyhow, it's the last time for the foreseeable future that this kind of breakage will happen. So after this is done, we are clear. :)

[BeatLink]

Im creating a personal image that rolls the server, web client and dav adapter all into one, and I'll use Nginx as the frontend server and proxy between the 3. So accessing my server on the etesync port will show you nginx with the main page being the webclient. the /admin page will take you to the server admin and /dav will take you to the dav adapter

[tasn]

That's very cool! I've been thinking about doing something like that, please share it once you do it, and maybe make each component configurable? I'm sure others would like it too. Maybe call it etesync-omnibus or etesync-bundle or dunno.

[BeatLink]

Sure thing. Doesnt have much configuration yet, im just trying to get it built. But its here if you want to look at it!

https://hub.docker.com/r/beatlink/etesync-omnibus

[tasn]

Cool! I'd maybe call it etesync- rather than etebase-. The server is etebase, but all of the rest of the stuff are etesync specific.

[BeatLink]

ohh. Okay

[BeatLink]

I think the name is fixed though. Id have to delete and recreate the repo

[BeatLink]

Shouldnt be long hold on

[BeatLink]

If you do create an official omnibus package you should use the etesync user rather than mine though hahahah.

[BeatLink]

I've been doing some reading and it seems like the best way to utilize docker is to use a docker compose file and multiple images to build an isolated environment, instead of all the components in a single image. So I will be working on that today

[BeatLink]

Bonus points is, we can reuse existing docker images so it should be much much more simple!

[victor-rds]

@victor-rds do you intend to update your images to 2.0?

The latest, slim and alpine tags are already available, the 0.5.1 variants are building right now, you can see the progress here: victor-rds/docker-etebase/actions

All tags are available as victorrds/etesync or victorrds/etebase, they are same build just different names

All tags are build for x64, ARMv6, ARMv7 and ARM64 (AArch64),

I've added some docker-compose examples here: victor-rds/docker-etebase/examples

This is the first batch of Etebase (Etesync 2.0) public images, so please if find any issues feel free to open in my repo

[tasn]

@victor-rds: amazing, good job!

[BeatLink]

Awesome!

[BeatLink]

I've been having some issues writing the omnibus package. The compose file requires a bit more configuration than thought. Namely there is no way to set the home server for the web client without building the files from scratch

[tasn]

Yeah, it's needed to be passed at build time (as it's just a static JS app), so there's really no way to do it in a generic pre-built image. Though I don't think putting in the server URL is such a big deal.

[BeatLink]

Ok. The best plan I can see is writing a tiny dockerfile to build and serve the web frontend and building that in the main omnibus docker compose

[tasn]

You can potentially just sed the js file to replace the server url before serving. So when BUILDING set the server url to @SERVER_URL_GOES_HERE@ and just replace it.

Anyhow, let's talk about the omnibus image elsewhere, this is not the right ticket for it.

[BeatLink]

Ok, where should I have discussion? A new issue?

[tasn]

I assume you have a repo for this, no? Maybe open a ticket there and CC me so I can participate? If you don't have a repo, etesync-web would be a more suitable location.

[SecT0uch]

@tasn Can we consider victor-rds's images as official ? @victor-rds Do you have CI/CD to automatically build and release an image when a new version is available ? Also victorrds/etesync and victorrds/etebase seem to be mirrors. Should I rather use victorrds/etebase ?

Thanks for you work all !

[tasn]

Sure, though we would need to move it under the etesync/ namespace for that.