AppImage / AppImageKit

Package desktop applications as AppImages that run on common Linux-based operating systems, such as RHEL, CentOS, openSUSE, SLED, Ubuntu, Fedora, debian and derivatives. Join #AppImage on irc.libera.chat
http://appimage.org
Other
8.78k stars 563 forks source link

Do releases #849

Open probonopd opened 6 years ago

probonopd commented 6 years ago

The lack of formal releases is cited as a reason for poor support of the AppImage format in distributions (like Fedora), desktops (like GNOME), and IDEs (like GNOME Builder).

You guys have made far too many choices favoring agility without any security.

Hence, we should do formal releases.

How can we do so in a way that does not create too much additional administrative work for us?

What should we formally release?

cc @Conan-Kudo

Conan-Kudo commented 6 years ago

I'd prefer to see pretty much everything have formal releases with tarballs and such (especially given the heavy use of submodules). Ideally, the regular releases means that submodules aren't as necessary since the dependencies can be expressed more normally in CMake.

probonopd commented 6 years ago

Yes, tarballs are automatically generated by GitHub when one does a tagged release.

Ideally, the regular releases means that submodules aren't as necessary since the dependencies can be expressed more normally in CMake.

How would releases enable this? Specifically, say we release once a month. We still want continuous versions of our own dependencies in our continuous builds. It's fine though that others (distributions) would use the monthly releases of all of our stuff, though.

Would that satisfy your requirements?

probonopd commented 6 years ago

Now, what's the easiest way to get Travis CI to do automated time-based releases, say, on the 1st of each month, without any manual intervention? https://docs.travis-ci.com/user/cron-jobs/

azubieta commented 6 years ago

IMHO, doing releases is a required thing. But it implies more administrative effort. You will have to maintain at least a master branch (which should be ultra stable) and a development branch (which will be used for our development job and will have continuous builds). Merging into master should never be automated. Bugs can sleep into the stable branch in the final second.

Which tools should get versioned this way? The ones that provide a public API, it doesn't matter if it's a CLI or a C header it would be required in order to inform our users. linuxdeploy will be a special case as it uses the AppDir spec as API with its plugins.

TheAssassin commented 6 years ago

@probonopd for linuxdeploy, it's already planned to provide releases in the future. We'll do semantic-like versioning in intervals, e.g., every 1-2 weeks. That way, we have a widely known, comprehensible release scheme, but don't add too much maintenance overhead. Patch versions will only be released to fix critical bugs. which cannot wait until the next release date.

Conan-Kudo commented 6 years ago

You will have to maintain at least a master branch (which should be ultra stable) and a development branch (which will be used for our development job and will have continuous builds).

This is absolutely not required. In nearly all of my projects, I deliberately avoid this workflow because it complicates things too much. Tagging off of the main branch that code is accepted in is perfectly fine, especially since you more or less operate in a fork+pull model.

TheAssassin commented 6 years ago

You will have to maintain at least a master branch (which should be ultra stable) and a development branch (which will be used for our development job and will have continuous builds).

This is absolutely not required. In nearly all of my projects, I deliberately avoid this workflow because it complicates things too much. Tagging off of the main branch that code is accepted in is perfectly fine, especially since you more or less operate in a fork+pull model.

Exactly. I don't want to do that either. I plan to tag off the master branch, too.

probonopd commented 6 years ago

IMHO, doing releases is a required thing.

Define "required". We did fine without this for the first decade of the project, by keeping all of AppImage in just one repo and doing nothing but continuous releases.

The complexity comes in since we now have a library that outside users might start using.

But it implies more administrative effort.

If I don't one thing in my hobby projects, it is administrative effort. In fact if my hobby comes with administrative effort then I will choose other hobbies ;-)

You will have to maintain at least a master branch (which should be ultra stable) and a development branch (which will be used for our development job and will have continuous builds).

This is dangerously going into traditional software engineering (great for larger projects and companies where people are paid to do jobs) and exactly the opposite of where I want to go (a hobby project that is fun to hack on in our free time).

I want this to be continuously released, the master branch needs always to be shippable, everyone (users and developers) use the same continuous version, it must never break (and if it does it is rolled back immediately, reverting any questionable commits). I don't want to deal with anything else but the master branch. This means that we need a certain level of test automation.

I also have absolutely no intentions to maintain outdated versions that may end up in Debian stable etc. over time. This is why I am not keen on having this stuff in distributions in the first place. Ideally we would only get this into debian backports (so that debian stable users can get the latest AppImage stuff) but I guess this is not how distros work. Which is why I invented AppImage in the first place ;-)

Merging into master should never be automated. Bugs can sleep into the stable branch in the final second.

Let's phrase it differently: Our master branch (which is always kept in a shippable state) should be automatically tagged with version numbers on a time-based schedule. It must not create additional work. I am already unhappy with the complexity that came with splitting what was initially one repository (where you could find every bug and code neatly in one location) over many different repos.

shoogle commented 6 years ago

The complexity comes in since we now have a library that outside users might start using.

Sounds like you should start by doing releases only for the library, and only think about doing releases for the other components once you have got the hang of it.

shoogle commented 6 years ago

Our master branch (which is always kept in a shippable state) should be automatically tagged with version numbers on a time-based schedule.

I don't think automatic tagging is possible, but once you create a tag and push it to GitHub, Travis can detect this and upload a new release. So creating the tag is the only manual step, and this is not much effort and can be scripted.

I suppose you might be able to automate the process if you take advantage of the fact that Travis is already building commits anyway. On each build you can have Travis check when the last tag was made, and have it create a new one if it has been more than X weeks since the last one (or only if there have been more than Y number of commits since then). It's not quite the same as a scheduled build, but it gives the same result in the end.

Calinou commented 6 years ago

I don't think automatic tagging is possible

It is possible to create tags on repositories using the GitHub API, see uploadtool :slightly_smiling_face:

shoogle commented 6 years ago

@Calinou, it wasn't the ability to create tags that I doubted, but the ability to do a scheduled build on Travis. However, I now see that this is also possible.

TheAssassin commented 6 years ago

Automatic tagging is a horrible idea. Let the developers decide which commit is stable enough to be released. Otherwise, half-finished or buggy work might possibly released.

probonopd commented 6 years ago

Otherwise, half-finished or buggy work might possibly released.

Well... Half-finished work should ideally never be pushed to the master branch. Buggy work should ideally never pass the test automation.

Eveything else means more admin work for someone (not me, as this is my hobby, and I don't like to do admin work in my free time).

shoogle commented 6 years ago

The only admin work is deciding which commit to tag. Once you know which commit you want to tag you can run a single terminal command to create the tag and push it to GitHub, and then Travis would build a binary release automatically.

Set yourself a calendar reminder every month to remind you to create a tag. If the code is buggy when it's time for a release, simply tag the last commit before the bug was introduced, or delay the release until the bug is fixed.

TheAssassin commented 6 years ago

We could even tag using the GitHub web interface.

probonopd commented 6 years ago

Looks like we have code supporting this in uploadtool already:

https://github.com/probonopd/uploadtool/blob/6729efc1dca9397583633c24eddb4372988fe284/upload.sh#L32-L37

probonopd commented 6 years ago

Started writing a script that automates this but seemingly a Travis job can't push to git by default, what am I missing? Do we need to use the GitHub API and a GitHub token for this? Any volunteers to complete https://github.com/AppImage/libappimage/blob/master/travis/tag-using-cron.sh? (I don't want something that creates recurring manual work but am looking for a "set once and forget" solution.)

To be honest, I am still not convinced that doing releases is beneficial for us, as every commit is a "release" in our continuous-release philosophy. All this does for us is

TheAssassin commented 6 years ago

I'm strongly opposed to auto-tagging. That doesn't make any sense, and is counterproductive.

probonopd commented 6 years ago

Then "do releases" means additional admin work for us, which I want to avoid (and won't be doing). I am not sold onto the idea anyhow. Our project is all about continuous releases, upstream distros should just treat each git commit to master as a "release".

Conan-Kudo commented 6 years ago

Then you're not actually solving the problem and once again you've proved my point about the double standard.

Your tooling is effectively "platform" and "infrastructure". That means you have a responsibility to stabilize it and provide predictable and useful releases so that they can be integrated and used to support applications and services.

If you don't want to have that kind of responsibility, just shut the AppImage project down and walk away.

Flatpak is doing a good job of this and is an excellent standard of how this should work.

Conan-Kudo commented 6 years ago

By the way...

How would releases enable this? Specifically, say we release once a month. We still want continuous versions of our own dependencies in our continuous builds. It's fine though that others (distributions) would use the monthly releases of all of our stuff, though.

Git submodules do not auto-update. If you're making formal releases of your dependencies, instead of relying on submodules, you can use CMake's native find_package() (or pkg_check_modules() with pkgconfig) mechanism to locate them.

This effectively defines the minimum version standard for compatibility and allows all the components to update at their own lifecycles.

TheAssassin commented 6 years ago

As a maintainer of various software, I have some experience in that topic. Not as much as other people, but I think I can evaluate the costs and benefits of different versioning types. I have done continuous deployment, time-based releases and stable/version branch stuff. My suggestions in that topic are based on that experience.

In my opinion, it will be a lot more work to have to maintain yet another script that somehow is bound to proprietary infrastructure (Travis & GitHub). We have some of these scripts, and they cause trouble all over the place. Error checks are missing or cannot be implemented properly, sometimes one build works and uploads its files, whereas the other one is failing, so architectures are missing, etc. That's quite annoying. The only way to solve these problems is usually to self-host a service that interacts with both other services, allowing to implement all the business logic properly. However, within AppImage, that's usually not a viable strategy.

In fact, we already have all we need to make "timed releases". Just push a tag and Travis will do the rest. For me it's not a problem to add some entry to my calendar to create those tags. Automating this just creates a lot more issues than it solves, human interaction is required for versioning.

Also, releasing can help improving software, as one needs to think about changes more than once. One needs to make sure not to break the own or someone else's software, or if they do, document that properly. That's something we used to lack anyway, but we're improving that recently. I'm not at all opposed to versioning, as long as it makes sense. For linuxdeploy, users request it, as corporate environments need that for a whole lot of reasons (including internal reviews (security or something), silly politics, etc.). Versioning linuxdeploy in this case is a great idea: we can ship our continuous software under the hood of versioned linuxdeploy releases. For instance, we ship the latest continuous release available of appimagetool in the releases of linuxdeploy, etc.

For our tools I don't want any versioning either. That was one of the main reasons to extract libappimage -- we can version that properly, improve the code, make the API stable, etc., and at the same time keep the continuous release scheme tools using that library (AppImageLauncher, appimagetool, ...).

Your main argument is always "work". Well, of course it's work. Things don't happen magically. But see it like this: you have a team of people willing to do that work, but your own politics sometimes prevent us from bringing AppImage forward. We managed to e.g., get KDE to develop their own AppImage specific technologies by just improving our library and providing it to them separately (which was great anyway, as the code used to be messy and buggy).

@Conan-Kudo is totally right that we already have some "manual" update work in the case of submodules. However, for us this supports our continuous release scheme. We only update dependencies when it's necessary, that way we don't have to continuously update our own software when we introduce breaking changes into internal libraries. Thinking about versioning in libappimage, there's not much that changes. In fact, we're improving our "bad" behavior when introducing versioning in libappimage in the future. We need to stabilize our APIs. And we just update our submodules to the latest release commits then. Not much difference to now.

probonopd commented 6 years ago

Hm. @Conan-Kudo you are making thoughtful points, which I appreciate and why I always like to discuss with you.

Then you're not actually solving the problem and once again you've proved my point about the double standard.

The AppImage project is about creating decentral, distribution-agnostic agile software distribution concepts that allows application authors to get out their software to end users without going through the hell that is distribution packaging with all the rules, policies, waiting times, etc.

The AppImage project does this by providing application authors with tools to make AppImages. Since these tools are "applications" too, the same rules (getting out or stuff directly to our users without going through the hell that is distribution packaging with all the rules, policies, waiting times, etc.) should apply to us, too.

Your tooling is effectively "platform" and "infrastructure".

No, the target system (=GNU/Linux/Xorg/glib/Gtk+/...) stack is the platform. Platform code needs to be stable, make guarantees, and not change (without prior notice). I would not say that our code has reached (or will ever reach) that level.

That means you have a responsibility to stabilize it and provide predictable and useful releases so that they can be integrated and used to support applications and services.

I am getting your point, and it's not without merits, only that we (or, more precisely, I) don't really want to do this. We publish AppImageSpec and ideally the desktop (GNOME, KDE, XFCE,...) people would take and implement these specs to increase usability. Like the people who wrote the ELF spec certainly did not write/maintain all ELF implementation in use today themselves. I have never said we should write _systeminfrastructure software (as the AppImage project) that other projects should build into their own solutions.

If you don't want to have that kind of responsibility, just shut the AppImage project down and walk away.

Again, I can see where you are coming from, and I would be even a bit more extreme: If we, as the AppImage project, would not "eat our own dogfood", then we should shut the AppImage project down and walk away. To be concrete, if we would not release our tools as AppImages but would need to rely on traditional software distribution methods...

probonopd commented 6 years ago

In fact, we already have all we need to make "timed releases". Just push a tag and Travis will do the rest. For me it's not a problem to add some entry to my calendar to create those tags.

Exactly.

Automating this just creates a lot more issues than it solves, human interaction is required for versioning.

Really? Scheduling something in a calendar just to go through a series of manual repetitive steps sounds wasteful to me and it's just not something I enjoy to do in my free time.

Also, releasing can help improving software, as one needs to think about changes more than once. One needs to make sure not to break the own or someone else's software, or if they do, document that properly.

That's the bigger point here, imho. We should strive for (I am not saying I am always perfect at it) to only commit stuff that works, and if it doesn't, revert it immediately. All other work ceases immediately if master is broken (does not build or creates showstopper issues when being used). Since we "eat our own dogfood" everywhere (i.e., use continuous builds of our tools ourselves) we are usually the first ones to encounter such issues, which should give us room to fix or rollback.

I'm not at all opposed to versioning, as long as it makes sense. For linuxdeploy, users request it, as corporate environments need that for a whole lot of reasons (including internal reviews (security or something), silly politics, etc.).

Couldn't those corporations mirror a certain commit (binary) and have the same advantages? Only that they would have to refer to it as, e.g., 8c8516f, rather than 3.20180806.0.

For our tools I don't want any versioning either. That was one of the main reasons to extract libappimage -- we can version that properly, improve the code, make the API stable, etc., and at the same time keep the continuous release scheme tools using that library (AppImageLauncher, appimagetool, ...).

Yes, having a thing such as libappimage brings in responsibilities (and expectations) that we didn't have before, when all of our code was only used by our own tools and we were free to change anything at any time. With libappimage, this is no longer the case, as @Conan-Kudo (rightly!) points out. Which is why I am looking for a way to prevent people using libappimage in other way other than linking to it statically, to ensure it is never seen as something stable that can be relied on, but merely as something you have to take care of delivering in the corresponding version yourself if you use it. (I don't want to see or support today's libappimage 5 years from now when it is in Debian oldstable!)

Your main argument is always "work". Well, of course it's work. Things don't happen magically.

Don't get me wrong, the whole project is "work", but creative, not administrative. For me this is purely a hobby, and I simply don't like to do unpaid administrative work.

But see it like this: you have a team of people willing to do that work

That's great. Just keep in mind:

So, in summary: If someone is stepping up to do the work, I am the last person to stop them. Just think about the implications, and don't expect me to do repetitive administrative work.

TheAssassin commented 6 years ago

Yes, in the end it's a lot less work, as fixing scripts and issues created by scripts is something super annoying (we have these bugs every couple of weeks in AppImage, and they take me hours to analyze and fix usually...). Clicking a button in a project after thinking about it for 2 minutes whether it's a good idea to release that commit is a lot less time intensive.

don't expect me to do repetitive administrative work.

Nobody does. And nobody guarantees the development of libappimage etc. anyway. If someone doesn't maintain the software any more, there is no point in creating releases any more.

probonopd commented 6 years ago

If someone doesn't maintain the software any more, there is no point in creating releases any more.

You have a point there 👍

develar commented 6 years ago

At least, thanks for Release build (11) :) It crashes for me, but at least now I am aware that electron-builder should update AppImage runtime because now it is clear that currently used is outdated. As was already stated, without formal releases I don't know should I update or not. And without changelogs not clear for me for what I should spend my time and update. But changelogs it is another topic.

probonopd commented 6 years ago

At least, thanks for Release build (11) :) It crashes for me

Hi @develar, have you opened a ticket for it?

As was already stated, without formal releases I don't know should I update or not.

Since electron-builder is only using the runtime, I expect there won't be frequent changes, and generally the latest continuous build should be the best one, as we don't do maintenance branches.

TheAssassin commented 6 years ago

@probonopd did you notice that we're doing releases in libappimage now? That's been a requirement for Debian packaging. It's no real work, just tag and push, as there's no release binaries uploading or so.

probonopd commented 6 years ago

If it's no work, then I ask someone to do it :+1:

TheAssassin commented 6 years ago

Nobody else expected this to be real work (https://github.com/AppImage/AppImageKit/issues/849#issuecomment-419732854).