search

MesserLab / SLiM

SLiM is a genetically explicit forward simulation software package for population genetics and evolutionary biology. It is highly flexible, with a built-in scripting language, and has a cross-platform graphical modeling environment called SLiMgui.
https://messerlab.org/slim/
GNU General Public License v3.0
160 stars 30 forks source link

SLiM 4.2 release process #434

Closed bhaller closed 5 months ago

bhaller commented 5 months ago

Hi folks! SLiM 4.2 is now happening. It has been tagged in git (v4.2). I'm editing this initial comment to have the usual checklist:

Onward and upward!

bryce-carson commented 5 months ago

There is an improvement we can make to RPM packaging since we merged my changes to the CMake file, and these changes seem to be working well (no issues reported!). Copr can use webhooks to trigger builds, so rather than me having to manually trigger a new build and indicate the Git commit hash of the new tag in that build, GitHub will automatically trigger a new build of the RPM package when you make a new tag.

Here are lightly modified instructions from the Copr documentation on how to add the webhook URL I am emailing you:

GitHub

How to use it:

  1. In your GitHub project, go to Settings / Webhooks
  2. Click on the Add webhook button.
  3. Fill in the Payload URL field with the URL emailed to you by @bryce-carson
  4. Select application/json as the content type.
  5. You do want to only react to Tag push events, so click "Let me select individual events," and then select Branch or tag creation, and be sure to un-select any other individual events (such as individual commits, branches, etc.; if fine-grained options are provided, select "Tag creation on the master branch").
  6. Click the Add webhook button.

Once you accept the PR I made, which updates the version in the RPM spec file, the only thing left for the RPM to be updated and become publicly available would then be to make a new tag. I won't need to navigate to Copr and trigger the build myself.

bryce-carson commented 5 months ago

Hi, @bhaller.

If you add this pre-commit hook to your clone of SLiM, you'll be informed that you need to update SLiM.spec when you're updating SLiM.pro with the Qt IDE. These are the files I've identified that should only be changed by you when you're preparing a new release, and which won't change otherwise (other files contain global variables, and may change between releases).

The place to put this script is .git/hooks/pre-commit within your clone of SLiM on your workstation, whichever one you make releases from (from the reading I did, it looks like you can't otherwise share client-side git hooks by pushing them in the repository). You'll need to delete .git/hooks/pre-commit.sample.

You must make the script executable with chmod +x, or Git will ignore it entirely and it will have no effect upon the commit process.

if [[ $(git diff --exit-code HEAD -- SLiM.pro) && $(git diff --exit-code HEAD -- SLiM.spec) ]]; then
  echo -e "\e[32;1mSUCCESS\e[0m";
  echo "The release process may now continue with tagging on GitHub."
  exit 0
else
  echo -e "\e[31;1mFAILURE\e[0m";
  echo -e "Ensure that \e[1mboth\e[0m SLiM.pro and SLiM.spec have been updated.\nSLiM.pro is updated in the Qt IDE, likely, and SLiM.spec can be updated with any\ntext editor."
  exit 1 
fi

The script is designed to warn you, and also prevent you from making a commit, when there are changes to one of the files, but the other files haven't changed. All the files need to change (as compared to the HEAD of the cloned repository) within the same commit.

This should enable you to slightly modify your workflow, such that you can be reminded to update the SLiM.spec file yourself, and then do whatever else needs doing with the other package maintainers, and then when you finally make a new tag on GitHub you'll automatically trigger an updated build on Copr (that is, assuming you add the webhook above, and want to adopt the increased automation).

I still welcome being tagged, as I also welcome having the latest version of SLiM installed on my machine---hot off the press---and it doesn't hurt to build from sources before "my users".

As a final note, if you adopt this before you accept the PR I made, you won't be able to make any commits if you change SLiM.pro. You might run into some roundabout trouble if you do this, so I recommend accepting the PR, fetching the changes from GitHub, and then adopt the commit hook (if you want).

bhaller commented 5 months ago

Hi @bryce-carson. I have a couple of questions. (1) SLiM.pro does not normally change for a new release. What change do you have in mind for me to make there? QtSLiM/QtSLiM.pro does change, since it contains the version number used by the QtSLiM build; is that what you meant to key off of? That would make more sense to me. (2) The trigger event for the webhook is "Branch or tag creation". But I sometimes create tags and branches for other reasons, and we don't want those to trigger a new build, right? Would it make more sense to have the trigger be "Releases"? Or is there some part of this scheme that will make the new Copr build actually only trigger when the version number has been incremented in the spec file or something? (3) It would be good to have this scheme documented somewhere more permanent than this issue, for posterity. Do you think you could add something to your PR, which I have not yet accepted, that adds some documentation about the Copr build process? Maybe in the CONTENTS file, unless you have a better idea? We don't really have a place to put internal developer documentation right now, hmm. Feel free to create a new file, maybe a .md file like DEVELOPER.md (or plain text if you prefer), in the top-level directory to make a place for internal developer notes, if that seems like a good path to you. Right now it would only contain whatever notes you might place there regarding the Linux release process, but I can imagine putting a couple of other things there too, including perhaps my own release process notes.

bryce-carson commented 5 months ago

Hi Ben,

Yep, you're right. I meant QtSLiM/QtSLiM.pro. If you change the filename in the script to that path it will work as intended.

Also, yes, "Releases" is the option we'd like to use to trigger a build for this webhook. There's no other scheme to specify on the Copr end that it should check the contents of the repository, but that's an interesting idea and I'll make a suggestion for a feature on the Fedora infrastructure comms. channels.

I can document this (and the RPM packaging), and I'll invite the other package maintainers to document their processes as well. The PR only changes the version string in the RPM spec file, and does not otherwise change anything on the SLiM or RPM end of things; what I have changed is the build settings for the package on Copr. The package is now set to build automatically, but that will only be triggered by the integrations (GitHub or other host webhooks). I can include the initial documentation in the same PR, if you'd like. I'll start documenting it now since it is fresh in my mind.

Regards,

Bryce.

On Thu, Mar 21, 2024 at 7:29 AM Ben Haller @.***> wrote:

Hi @bryce-carson https://github.com/bryce-carson. I have a couple of questions. (1) SLiM.pro does not normally change for a new release. What change do you have in mind for me to make there? QtSLiM/QtSLiM.pro does change, since it contains the version number used by the QtSLiM build; is that what you meant to key off of? That would make more sense to me. (2) The trigger event for the webhook is "Branch or tag creation". But I sometimes create tags and branches for other reasons, and we don't want those to trigger a new build, right? Would it make more sense to have the trigger be "Releases"? Or is there some part of this scheme that will make the new Copr build actually only trigger when the version number has been incremented in the spec file or something? (3) It would be good to have this scheme documented somewhere more permanent than this issue, for posterity. Do you think you could add something to your PR, which I have not yet accepted, that adds some documentation about the Copr build process? Maybe in the CONTENTS file, unless you have a better idea? We don't really have a place to put internal developer documentation right now, hmm.

— Reply to this email directly, view it on GitHub https://github.com/MesserLab/SLiM/issues/434#issuecomment-2012299243, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMMRAZJCIHLHPSGENLRFXI3YZLODTAVCNFSM6AAAAABFANTMHWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJSGI4TSMRUGM . You are receiving this because you were mentioned.Message ID: @.***>

bhaller commented 5 months ago

Hi @bryce-carson. OK, thanks. I haven't accepted the PR, so I think you can make the QtSLiM/QtSLiM.pro change with your other revisions to the PR? Let me know when it is ready to go. I will set up the webhook to trigger on "Releases", not "Branch or tag creation", if I now understand correctly. :->

HOWEVER: The drawback of triggering on "Releases", thinking things through, is that the release will be declared and public by the time the Copr build kicks off. If the Copr build fails for some reason, I guess a new release will need to be tagged to fix the problem. That was the nice thing about (1) tagging, (2) doing builds based on the tag, and then (3) declaring the release once the builds succeeded. I'm not sure how much I care about this problem in practice. The Windows pacman build already has to wait for the release to be tagged before its crank can start turning, so it already has this problem; and I can do the Conda build with just the tag, myself, and make sure that succeeds before I make the release.

I could make the release but not "publish" it; that's an option on GitHub. It looks like the "Releases" webhook would trigger once when the release is made, and then trigger AGAIN when the release is "published". Will that pose a problem of any kind for this workflow?

Let me know what you think about all that. I think the process should work ok, if you prefer it to be this way. Thanks!

bhaller commented 5 months ago

Not sure whether editing the comment at the top re-notified you folks; perhaps not. So, please see the new top comment. We're rolling! @bryce-carson @grahamgower @rdinnager

bryce-carson commented 5 months ago

Thanks for keeping us apprised.

Editing comments does not renotify people, I think.

On Thu., Mar. 21, 2024, 2:51 p.m. Ben Haller, @.***> wrote:

Not sure whether editing the comment at the top re-notified you folks; perhaps not. So, please see the new top comment. We're rolling! @bryce-carson https://github.com/bryce-carson @grahamgower https://github.com/grahamgower @rdinnager https://github.com/rdinnager

— Reply to this email directly, view it on GitHub https://github.com/MesserLab/SLiM/issues/434#issuecomment-2013709815, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMMRAZP5SNYY5R2WGDNJFJTYZNB5BAVCNFSM6AAAAABFANTMHWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJTG4YDSOBRGU . You are receiving this because you were mentioned.Message ID: @.***>

bryce-carson commented 5 months ago

HOWEVER: The drawback of triggering on "Releases", thinking things through, is that the release will be declared and public by the time the Copr build kicks off. If the Copr build fails for some reason, I guess a new release will need to be tagged to fix the problem. That was the nice thing about (1) tagging, (2) doing builds based on the tag, and then (3) declaring the release once the builds succeeded. I'm not sure how much I care about this problem in practice. The Windows pacman build already has to wait for the release to be tagged before its crank can start turning, so it already has this problem; and I can do the Conda build with just the tag, myself, and make sure that succeeds before I make the release.

I could make the release but not "publish" it; that's an option on GitHub. It looks like the "Releases" webhook would trigger once when the release is made, and then trigger AGAIN when the release is "published". Will that pose a problem of any kind for this workflow?

Let me know what you think about all that. I think the process should work ok, if you prefer it to be this way. Thanks!

🤠 Howdy.

Those are fair criticisms, but I think the rarity of a build issue that is actually related to Fedora should belay most of the concern. We've only had a single issue with building on Fedora, and that was with the robinhood library. I can always build from source on my own machine before we release.

I don't mind flicking the switch manually, it's just another option (how many have I thrown at you now?) for a little automation.

In this instance, I'll throw the switch manually, and there's nothing you need to do unless you want to adopt the automation before the next (point?) release.

bryce-carson commented 5 months ago

@bhaller All good here, Ben. Go ahead and check off my installers from the task list.

https://copr.fedorainfracloud.org/coprs/bacarson/SLiM-Selection_on_Linked_Mutations/build/7194959/

bhaller commented 5 months ago

Let me know what you think about all that. I think the process should work ok, if you prefer it to be this way. Thanks!

🤠 Howdy.

Those are fair criticisms, but I think the rarity of a build issue that is actually related to Fedora should belay most of the concern. We've only had a single issue with building on Fedora, and that was with the robinhood library. I can always build from source on my own machine before we release.

Yes, probably rare; but it would suck when it happened. :-< Perhaps building on your own machine would alleviate much of the worry; I can add that to my standard release checklist.

I don't mind flicking the switch manually, it's just another option (how many have I thrown at you now?) for a little automation.

Yeah; honestly the automation is more conceptually complex than just doing it un-automated! (Especially given that you're willing to help with it, but perhaps even if you weren't.) :->

In this instance, I'll throw the switch manually, and there's nothing you need to do unless you want to adopt the automation before the next (point?) release.

Thanks!

bhaller commented 5 months ago

Hi @bryce-carson. The AppImage install is still non-functional, correct? Shall I just remove the mention of it in the manual? Or edit that bit to be grayed out or something, indicating it is non-functional? I'm concerned that describing an installation method that does not, in fact, work might lead to confusion. Do you concur? And when the user tries to follow the instructions in the manual for AppImage, what happens? I.e., what's the failure mode here?

EDIT: Well, if it was good enough for 4.1, it's good enough for 4.2, I suppose; we must have discussed this before. The manual has been put to bed and the release is rolling, so never mind. But let's think about this for the next release; if AppImage isn't happening, let's just pull the plug on it. Doesn't break my heart, since it would install only SLiMgui anyway; seems like perhaps more trouble than it's worth.

bhaller commented 5 months ago

Ping @grahamgower re: Arch installer. :->

bhaller commented 5 months ago

@rdinnager, the new release is now published on GitHub, so I think you can start the pacman crank turning. Let me know when it finishes. :-> Thanks!

bhaller commented 5 months ago

The release has been announced: https://groups.google.com/g/slim-discuss/c/TbjnwoUjdyE

bryce-carson commented 5 months ago

Hi @bryce-carson. The AppImage install is still non-functional, correct? Shall I just remove the mention of it in the manual? Or edit that bit to be grayed out or something, indicating it is non-functional? I'm concerned that describing an installation method that does not, in fact, work might lead to confusion. Do you concur? And when the user tries to follow the instructions in the manual for AppImage, what happens? I.e., what's the failure mode here?

EDIT: Well, if it was good enough for 4.1, it's good enough for 4.2, I suppose; we must have discussed this before. The manual has been put to bed and the release is rolling, so never mind. But let's think about this for the next release; if AppImage isn't happening, let's just pull the plug on it. Doesn't break my heart, since it would install only SLiMgui anyway; seems like perhaps more trouble than it's worth.

Indeed. There has been some pitfall with each non-native solution I've tried (Flatpak, AppImage). The BASH script has worked well for every user and it's extremely robust. Unless dpkg changes underneath our feet (Debian would never do that, I think) there's no reason for it to ever break.

I'm curious about the manual release process. Can we document this? I understand the manual is included with the legacy SLiMgui application you tend to maintain, and it is also shipped with QtSLiM.app. What about documentation fixes? Point-releases with only documentation changes?

Concerning the manual, I've also taken a little issue with the usability of the document given it's tight paragraphing. Perhaps we could add some extra spacing after paragraphs of the same style, and increase the indentation of the first-line. I think that would make it a little more readable.

grahamgower commented 5 months ago

Thanks @bhaller, the Arch installer is now updated.

bhaller commented 5 months ago

Thanks @bhaller, the Arch installer is now updated.

Thanks @grahamgower, sorry to interrupt your vacation!

bhaller commented 5 months ago

...

Indeed. There has been some pitfall with each non-native solution I've tried (Flatpak, AppImage). The BASH script has worked well for every user and it's extremely robust. Unless dpkg changes underneath our feet (Debian would never do that, I think) there's no reason for it to ever break.

Hey @bryce-carson. Yeah, the script seems fine. So, shall I delete the AppImage stuff in the manual for the next revision, then? Shall we consign it to the dustbin of history?

I'm curious about the manual release process. Can we document this? I understand the manual is included with the legacy SLiMgui application you tend to maintain, and it is also shipped with QtSLiM.app. What about documentation fixes? Point-releases with only documentation changes?

I'm not sure exactly what you're asking, but the process is something only I can do, at present. The manual is kept as a Pages file, which means (a) it's a big binary, doesn't diff, and can't live in GitHub easily, and (b) if more than one person was editing it there would be conflicts that would be very difficult to resolve (since, again, binary and can't diff). The manual snippets (only a small minority of the full manual) that are available in SLiMguiLegacy are converted from Pages format to RTF format (manually) whenever a change is made, and then for QtSLiM the RTFs in SLiMguiLegacy are converted into HTML. So, suffice to say it is a very complex process that is not ideal but is unlikely to change any time soon; it would be a LARGE project.

Concerning the manual, I've also taken a little issue with the usability of the document given it's tight paragraphing. Perhaps we could add some extra spacing after paragraphs of the same style, and increase the indentation of the first-line. I think that would make it a little more readable.

The format of the manual can't change, for better or worse. If you look at the PDF manual, you will notice that the page breaks generally come at opportune moments; even though there are large figures, tables, etc. in the text, the page flow somehow works out nicely. That is not a coincidence; a truly surprising amount of work has gone into ensuring that the layout of the manual in PDF is nice. If spacing, font, indentation, etc., were to change, all of that careful layout work would be destroyed. So... nope, sorry. Again, not ideal but unlikely to change any time soon.

rdinnager commented 5 months ago

@bhaller Sorry, been a bit busy this week. Should be able to get to this by end of tomorrow though. Thanks for your patience!

bhaller commented 5 months ago

@bhaller Sorry, been a bit busy this week. Should be able to get to this by end of tomorrow though. Thanks for your patience!

Hi @rdinnager. Life is busy, yes! It would be great if you could start the crank turn, since it takes a long time to finish; can that happen ASAP? Thanks!

rdinnager commented 5 months ago

@bhaller My PR was merged last night: https://github.com/msys2/MINGW-packages/pull/20511 . It has been built but is still queued for upload: https://packages.msys2.org/queue . Should be available for users soon hopefully. Cheers!

bhaller commented 5 months ago

@bhaller My PR was merged last night: msys2/MINGW-packages#20511 . It has been built but is still queued for upload: https://packages.msys2.org/queue . Should be available for users soon hopefully. Cheers!

Fantastic, thanks! :->

bhaller commented 5 months ago

Looks like the Windows pacman installer is up, so this issue is done and the 4.2 release is complete. Thanks folks!