search

backdrop / backdrop-issues

Issue tracker for Backdrop core.
144 stars 39 forks source link

Support dev/nightly (pre)releases #664

Open klonos opened 9 years ago

klonos commented 9 years ago

In d.org we had the notion of dev versions. At some point there was even versioning of core as well as contrib with the addition of a +z-dev to their version property in their .info files (for example version = "7.x-1.1+5-dev").

How do we intent to tackle this here?

klonos commented 9 years ago

...can we perhaps automate the generation of a +[latest_commit_timestamp]-git and append that to the version property in our .info files?

klonos commented 9 years ago

...asking because I started testing some of the backdrop contrib (no releases at the moment of writing this) and because the version property in their .info files is blank, that's what I got:

backdrop_contrib_modules_no_dev_versions

klonos commented 9 years ago

How this worked with drupal was that there was a single dev release for each branch. This dev release was being overwritten by each new one created (after a commit). The name was the same, but you'd still get update notifications if there was a newer one released (so, you knew if it was time to update).

quicksketch commented 9 years ago

...can we perhaps automate the generation of a +[latest_commit_timestamp]-git and append that to the version property in our .info files?

This all factors into making the Github service for Backdrop https://github.com/backdrop-ops/backdropcms.org/issues/8. Once we have a registered service, I believe we'll be able to make modifications to the code before they're packaged via Github's webhooks react to packaged releases. That would allow us to tack on the dev version string, as well as tack on the actual release version string in the event that that module is a tagged release.

Unfortunately, I don't think we could actually modify the archive Github creates. I think we only have the ability to possibly create and host our own archive. Perhaps this should all be discussed in https://github.com/backdrop-ops/backdropcms.org/issues/8

quicksketch commented 9 years ago

I believe we'll be able to make modifications to the code before they're packaged via Github's webhooks react to packaged releases.

It turns out, this is actually possible. Hooray! https://developer.github.com/v3/repos/releases/#upload-a-release-asset

klonos commented 9 years ago

Available updates warnings about 1.1.2 while using a 1.2.x-dev:

unknown_release_date-status_report

unknown_release_date-available_updates

quicksketch commented 8 years ago

Coming back to this after a long while. We implemented official packaged versions quite a while ago in https://github.com/backdrop-ops/backdropcms.org/issues/8.

This makes it so that when a developer creates a tag and a release, we can run a packaging script on it to insert the version information and packaging time. This works great.

What doesn't work great is the original purpose of this issue: Dev-release versioning. Because dev versions on GitHub aren't releases, we don't have the ability to attach a package to them. I'm not sure what to do about this. The only solution we'd be able to do immediately would be to self-host these archives on BackdropCMS.org (or somewhere other than GitHub).

In the mean time, I think we should pursue changes in our software to better handle the situation, such as https://github.com/backdrop/backdrop-issues/issues/1272.

klonos commented 8 years ago

Because dev versions on GitHub aren't releases, we don't have the ability to attach a package to them.

Yes, but (just thinking out loud here)... is it possible to somehow react on each commit and insert the time of the last commit in the .info files or something?

Anyway, we should be smart enough to not annoy the user with any warning if for example the installed version is 1.3.x-dev while the latest release is 1.2.2 (1.3.x > any 1.2.x).

quicksketch commented 8 years ago

Yes, but (just thinking out loud here)... is it possible to somehow react on each commit and insert the time of the last commit in the .info files or something?

I'm not sure we'd want to do this. We'd actually need to make another commit to GitHub in order to do this, modifying history of the files automatically. We might do this on a daily basis to minimize commits. Otherwise you may end up with hundreds of commits on the .info file just updating the timestamp line repeatedly.

So it is an option, though perhaps not a good one.

Anyway, we should be smart enough to not annoy the user with any warning if for example the installed version is 1.3.x-dev while the latest release is 1.2.2 (1.3.x > any 1.2.x).

Yep, that's part of the idea in https://github.com/backdrop/backdrop-issues/issues/1272.

klonos commented 8 years ago

Coming back to this annoyance, how about this idea...

GitHub has a way to know the time at which the last commit has happened. Right?:

backdrop-latest_commit_date

...and If I'm not mistaken, this can be retrieved per branch(?). So we can have this data for the latest devs 1.x (currently 1.3.x-dev), 1.1.x and 1.2.x. That's step one.

Now, as you have mentioned before, we cannot add the latest commit hash/date to a file because that would trigger another commit. Sure, but we can have a file with the last known commit before the current commit:

https://github.com/backdrop/backdrop/commits (this can also be filtered by branch!)

So, I think we can have a solution! Here's the plan:

  1. We add a file (.previous_commit or .version or something) where we can store a hash and/or date of a commit. This can be an existing file too if that's preferable.
  2. Before or with each commit we make, we make sure we update this file with the last known commit for the respective branch.
  3. We add a check that compares the file with the commit hash/date from step 1 to the previous to the current commit in GitHub (the respective branch of the installed dev version). If these are the same, it means that we had only a single commit after that (the current). If different, there there must be another, newer commit, so we can trigger "new version available".
  4. We find a way to automate step 2.

:tada:

This will allow us to also

quicksketch commented 8 years ago

Thanks for thinking on this @klonos! So if I'm understanding this, you're suggesting that we have a way to modify code in the repository itself (e.g. the .previous_commit file) so that when the dev release zip files are downloaded from GitHub, they already have this file in them. That would avoid us needing to self-host modified versions of these packages.

Before or with each commit we make, we make sure we update this file with the last known commit for the respective branch.

Unfortunately, the problem is right here. If we modify anything within the repository, it would have to be another commit. Essentially every commit (or more accurately every push to GitHub) would cause a second commit just to update this file. So we're still stuck in modifying the project history just to keep track of the packaging date.

klonos commented 8 years ago

So if I'm understanding this, you're suggesting that we have a way to modify code in the repository itself (e.g. the .previous_commit file) so that when the dev release zip files are downloaded from GitHub, they already have this file in them.

Yep, that was what I was suggesting.

Unfortunately, the problem is right here. If we modify anything within the repository, it would have to be another commit...

What I was suggesting here (IIUC how GitHub works) is that before we commit a PR, we make sure that in that very same PR the .previous_commit file is modified so that it includes the previous commit hash/timestamp. Since you are the one making the final commits of PRs, this would mean that this would add a step in your procedure of committing changes. Hence the need to somehow automate this.

quicksketch commented 5 years ago

A new idea here:

Our big hold-up is that we need a place to put a modified version of the code (one that has the .info file updated with the current version information).

We were discussing today on the weekly meeting that perhaps we could attach additional archives to the release pages on GitHub.

image

i.e. After "Download backdrop 1.13.3" add:

A new nightly release would be created every day in which commits occurred. However this would likely create a lot of links on the release page. As long as they were in the right order though, I don't think it would be too much of a problem. Especially as we encourage folks to use backdropcms.org for finding/downloading modules anyway.

There is no apparent limit on releases per version: https://help.github.com/en/articles/about-releases

klonos commented 3 years ago

Not sure if we should be creating a release every time there is a commit 🤔 ...that would cause "dev release flooding" (in both GitHub, as well as b.org).

Here's what I was thinking:

  1. At each given point, there would be a single dev version, as opposed to multiple dev releases on GitHub (this is how dev versions also work on d.org btw - only the most recent dev version is available at each given time). That dev version would effectively be the latest 1.x branch, in the form of a .zip (for which GitHub already provides an easy way to download, via https://github.com/backdrop/backdrop/archive/1.x.zip).
  2. We need to be able to add an identifier somewhere, and that cannot be in the system.info file or bootstrap.inc. Perhaps a new .version file at the top of the folder structure? 🤷 (that gets git-ignored).
  3. The addition of that identifier should be possible to be automated with GitHub pre- or post-hooks.
  4. The identifier could be a version string, or timestamp, or commit hash:
    • commit hashes make sense only to developers, and this group of users has a way and the knowledge to get this info.
    • a version string would require some logic that would read the current/previous version, and then properly increment it. As simple as this logic may be, it can be prone to errors/bugs, but the benefit is that humans would be able to read the version, and then be able to tell if it's newer/older than what they have. What they would not be able to tell is how old a specific version is (no date info can be derived from things like +1, +2 etc).
    • a timestamp of when the commit happened would also work, and very minimal if any at all logic would be required to generate them (no need to compare to previous version timestamp before generating a new, since they are basically integers that show time in seconds lapsed since Jan 01 1970, so they always go up). Comparing version numbers with unix stamps would be easy for machines/code, but humans would not be able to make anything out of it just by reading it. We could be converting the timestamp to a human-readable format, but then we'd need to decide things like yyyymmdd vs yyyyddmm, whether to include time at to which level of accuracy (because in theory, there could be 2 commits within the same minute), etc.

Not having specific GitHub releases would mean no release nodes in b.org, which would in turn mean that we'd need to poll this .version file in the GitHub repo. I don't think that this is an issue though.

With the release not living in b.org, it'd mean that the updater would need to be adapted to download from GitHub, which is OK, but unsure if there's anything special that our packaging script needs to do before a release is ready to be installed (I'm assuming that nothing).

One thing to note is that we'd need to make sure that the pre-/post-hooks updates this .version file in both 1.x, as well as the previous minor branch. So at the time of writing that'd be 1.x (which holds 1.19.x-dev), as well as 1.18.x (which holds 1.18.x-dev). This also means that for sites that have not be updated to the most recent minor, the updater would need to know how to download core from the appropriate URL, like https://github.com/backdrop/backdrop/archive/1.18.x.zip

The define('BACKDROP_VERSION', '1.18.x-dev'); in bootstrap.inc + this timestamp in .version should be enough to calculate which dev "version" is installed, as well as compare it to other dev "versions".

klonos commented 2 years ago

Not sure if we should be creating a release every time there is a commit 🤔 ...that would cause "dev release flooding" (in both GitHub, as well as b.org).

It seems that there are ways to work around that:

And there are ways to automate that with redy-to-use GitHub Actions:

klonos commented 2 years ago

@stpaultim ^^ 😉

image

stpaultim commented 2 years ago

Sorry, but this issue feels like it's closer to my specific concern. https://github.com/backdrop-ops/backdropcms.org/issues/236

But, thanks for pointing out this issue, which lead me to the other one. Maybe they are intertwined in ways that are not yet clear to me. :-)

klonos commented 3 months ago

Done some fresh research on this, and this looks promising: https://dev.to/derlin/how-to-create-nightly-releases-with-github-actions-fpp ...it uses a GitHub Action, and creates a single nightly release, which being updated with the artifacts of the latest successful build (older artifacts being automatically replaced by the new ones)

There's other GHAs to check as well, such as this one: https://github.com/marketplace/actions/nightly-release

  • Update Release name with simple template with date
  • Create/change nightly tag to head ref
  • Remove old assets, and upload new ones

Using GHAs though will put the burden on the contrib maintainers to implement on their repos. Can we automate/template that out for them?

klonos commented 3 months ago

...this one looks good as well: https://blog.goreleaser.com/nightly-builds-with-goreleaser-and-github-actions-e08c078e4d0f