search

cypht-org / cypht

Cypht: Lightweight Open Source webmail aggregator [PHP, JS]. Supports IMAP/SMTP, JMAP and soon EWS
http://cypht.org
GNU Lesser General Public License v2.1
1.01k stars 164 forks source link

Automate Docker builds (Especially from Master, but might as well do for all the releases) #1175

Open marclaporte opened 3 months ago

marclaporte commented 3 months ago

A sizable part of Cypht users install it via Docker. Thanks to @jonocodes via https://github.com/cypht-org/cypht-docker/issues/31, we now have: https://hub.docker.com/r/cypht/cypht

As of now, Docker releases (of Cypht stable releases) are manual. This is not a problem as we release stable versions every 2-3 months or so. However, for the development / testing / community process, it's causing quite a bit of friction. We need a way for community testers to get the latest Cypht from master. It could be a daily build, or even for each commit.

A nice side-effect is that will likely help us catch build bugs sooner.

Thoughts?

Thanks!

jonocodes commented 3 months ago

Ok, thinking out loud here....

We can use the date as a tag, but there is no reason to pollute dockerhub with tons of images. So yes, I do think the 'daily' version could be good here.

I think we should tag it 'nightly'. Not sure why, but I think that is a more commonly used/understood name? See https://www.mozilla.org/en-US/firefox/131.0a1/releasenotes/

So lets create a github workflow that does that every day. I think @wangxiaoerYah maybe can do this since he knows about github workflows. You can follow this for the tagging process: https://github.com/cypht-org/cypht/wiki/How-to-release-Cypht#manually-releasing-a-docker-image

To clarify every day an image will be built and pushed with the name 'cypht/cypht:nightly'. Thus overriding the previous day's build.

Separate items not discussed in this ticket:

  1. automating minimal CI, to make sure nightly actually boots at least
  2. automating production builds
marclaporte commented 3 months ago

I am OK with 'nightly' but 'daily' seems a bit better: https://en.wikipedia.org/wiki/Daily_build

marclaporte commented 2 months ago

https://github.com/cypht-org/cypht/releases/tag/v2.3.0 was released and now also on DockerHub: https://hub.docker.com/r/cypht/cypht/tags

jonocodes commented 2 months ago

v2.3.0 (release) was released and now also on DockerHub: hub.docker.com/r/cypht/cypht/tags

Great. Thanks @Shadow243

neotwix commented 2 months ago

Hello, Do you have plan to make a workflow for arm64 too ? sailfrog/cypht-docker contain one where the official not. thnaks for your work.

rodriguezny commented 2 months ago

Hello, Do you have plan to make a workflow for arm64 too ? sailfrog/cypht-docker contain one where the official not. thnaks for your work.

Do you mean builds for arm architectures ? Yes, we will add them too.

neotwix commented 2 months ago

Yes For the arm architecture. I should be Fine. Thanks

rodriguezny commented 2 months ago

Yes For the arm architecture. I should be Fine. Thanks

Ok, it will be added ASAP.

marclaporte commented 1 month ago

Ok, it will be added ASAP.

Please add manual instructions ASAP to ease testing.

rodriguezny commented 1 month ago

Hello, Do you have plan to make a workflow for arm64 too ? sailfrog/cypht-docker contain one where the official not. thnaks for your work.

Hello, I added a build for arm64.

Yes For the arm architecture. I should be Fine. Thanks

Ok, it will be added ASAP.

linux/arm64 added: https://hub.docker.com/r/cypht/cypht/tags

rodriguezny commented 1 month ago

Hello, Do you have plan to make a workflow for arm64 too ? sailfrog/cypht-docker contain one where the official not. thnaks for your work.

arm64 added https://hub.docker.com/r/cypht/cypht/tags, you can test it.

marclaporte commented 4 weeks ago

Re-opening as we don't yet have Docker builds from master. It would really smooth out our dev-test feedback loop, like here: https://github.com/cypht-org/cypht/issues/1153#issuecomment-2440240071

Also, for releases, the Docker part is manual. This is low priority as it's a manual operation that only needs to be done for stable releases (every few months).

@jonocodes: I remember you had a mental roadmap. Can you share some more wisdom?

Thanks!

marclaporte commented 4 weeks ago

ok, I found "After that we decide how to work this into a github action/automation for the next release perhaps. And how to maintain 'latest' and other non-versioned tags." here: https://github.com/cypht-org/cypht/pull/1001#issuecomment-2130022277

marclaporte commented 4 weeks ago

To clarify every day an image will be built and pushed with the name 'cypht/cypht:nightly'. Thus overriding the previous day's build.

I agree with the principle.

Reminder of our lifecycle: https://github.com/cypht-org/cypht/wiki/Lifecycle

Importantly, 2.x is supported for one year once 3.0 is released. Thinking of use cases, how about something like this?

So users must proactively determine a specific version, or latest stable release per branch, or daily build for master and major versions.

marclaporte commented 4 weeks ago
  1. automating minimal CI, to make sure nightly actually boots at least

So if tests fail, cypht/cypht:master-daily can be stuck to a few days ago. That is OK, as we'll fix it fast enough and it's less risky for users.

We already have automated tests for each merge requests before they are accepted in master. What would be different between daily build tests? Some longer tests?

marclaporte commented 4 weeks ago

@wangxiaoerYah: @SKB-TECH will now start working on this, so now is a good time for any suggestions.

Thanks!

jonocodes commented 4 weeks ago

Yes I think this sounds good. And I agree that having daily fall behind it tests break is the way to go.

I would go with this naming scheme as it seems more consistent with how others do it. However this is no real standard, so feel free not to.

cypht/cypht:2.4.0
cypht/cypht:daily
cypht/cypht:3-daily
cypht/cypht:2-daily
cypht/cypht:3
cypht/cypht:2

My one concern is '2-daily'. The issue is that eventually it will no longer be updated, but will still be called 'daily'. But this is a minor concern and can probably be cleared up with documentation.

marclaporte commented 4 weeks ago

@JohnXLivingston @mose @benoitg Any wisdom?

JohnXLivingston commented 4 weeks ago

@JohnXLivingston @mose @benoitg Any wisdom?

For what i understand, the main point of the daily build is to have some people test the most active branch. This is not meant for developpers that are backporting fixes from 3.x to 2.x. Those will not use docker to test, but their dev environment. If we have multiple daily version, i think nobody will use the old ones.

So, i think that only one daily build is enough, and I think it does not need to specify the major version in its name.

JohnXLivingston commented 4 weeks ago

Something else that could be usefull: having special tags pointing to the latest stable version.

When you have a docker environment (for example using docker-compose), you must specify the tag you want. Then, to update the software when there is a new release (for security fix for example), you have to do a docker-compose pull. So, when users are using tag as "2.4.0", the image won't update if the new version has a different tag ("2.4.1", "2.5.0", ...). Admins have to know there is a new version number, and have to change the configuration manually.

A common workaround is to have some special tags:

Those are just tags. Multiple tags can share the same build (no need to build X times, just build the new image, and change existing tags to point at it).

(i don't say that we must have all those tags, we just have to choose the preferred strategy)

benoitg commented 4 weeks ago

My only concern it that we should make sure not to create a naming convention ambiguity with the snapshots we want to create for tiki, which are human triggered quasi releases meant for dogfooding in production, vs automated daily builds.

We didn't iron out a naming convention either, but tiki snapshots should have the branch and date as part of their name.

Yes it's not the same project, but since it's mostly the same people, there is a very real risk of confusion.

But i'm not well versed enough on the assumptions of the various tooling to form an opinion on the exact naming convention we should use.

jonocodes commented 4 weeks ago

Something else that could be usefull: having special tags pointing to the latest stable version.

When you have a docker environment (for example using docker-compose), you must specify the tag you want. Then, to update the software when there is a new release (for security fix for example), you have to do a docker-compose pull. So, when users are using tag as "2.4.0", the image won't update if the new version has a different tag ("2.4.1", "2.5.0", ...). Admins have to know there is a new version number, and have to change the configuration manually.

A common workaround is to have some special tags:

* latest, which points to the latest stable version (see for example [nginx latest](https://hub.docker.com/_/nginx/tags?name=latest))

I do not like the use of 'latest' as it is ambiguous since it means different things to different docker systems/users. So I specifically avoid that term. In its place I think 'stable' may be appropriate. That being said, I dont encourage such use as a production user should at least know which major version they want to run. If 'stable' is pointing to 2, and there is a breaking change when it switches to 3, this will be problematic when the version gets changed out from under the user.

* have some tags like "develop" (equivalent to the "daily" we discuss here) and "production" (see [peertube](https://hub.docker.com/r/chocobozzz/peertube/tags))

Agreed. That is the intent of the above daily/nightly above. I generally think this is a case that should not come up since the included build tools make it unnecessary, but thats just me.

* have a tag with the exact version ("2.4.2") and another with the minor version ("2.4") that points to the last security patch version (see for example [nginx 1.27](https://hub.docker.com/_/nginx/tags?name=1.27))
* maybe same thing with "2" that points to the latest "2.x"

Yes. Those are the intent of the above mentioned 'cypht/cypht:2' which is would point to 'cypht/cypht:2.4.0'