BOS proposal: Continuous Integration (and the likes)

[pancetta]

We would like to set up a session on continuous integration containing e.g.

• what it is, how it works, examples
• Github Actions vs. Gitlab runners
• how do you do CI for HPC codes?
• what about continuous benchmarking?
• continuous integration or continuous testing?

Anyone interested in this?

[uekerman]

Something that is more and more important for us (meaning the research software preCICE): How to test over multiple repositories with downstream and upstream dependencies. @MakisH

[MakisH]

Yes, indeed, I would be very interested in contributing! :smile:

[pgierz]

I just proposed my own session for containers, but I would also be extremely interested in joining this group as well...

[led02]

• Providing dedicated / special executors (e.g., to use user-locked licensed software in the loop)
• CI and hardware-in-the-loop testing

[pancetta]

Thanks for all your input! Looks like we should do this session, then. But, well, how? Talks? Hands-on? Discussion? Let me hear what you think.

[MakisH]

Thanks for all your input! Looks like we should do this session, then. But, well, how? Talks? Hands-on? Discussion? Let me hear what you think.

I would be very interested in tutorial-like short talks, with some longer slot to try any of the presented tools hands-on, with the help of the presenter, on some pre-defined tasks. We would not be able to try everything, but on the other hand, we could choose and focus on the most relevant for each one of us.

Another aspect (maybe in parallel or after the talks) could be collaboratively creating a map of tools, with short description and common use cases. We could even compare/relate some of them (e.g., "Apptainer is like Docker, but you don't need root access").

[HeidiSeibold]

I added a label to this break-out. Can you check if you feel it is appropriate and change it if not? Let me know if you have any questions.

[pancetta]

So, it looks like we have plenty of items we could talk about. How about we have some opening talk, giving an introduction to the topic, and then we start to gather these items, make a wish list and decide how we could continue with this? Unless.. we actually find people willing to give short tutorial talk as proposed by @MakisH.

[jakob-fritz]

If you want to, I could give a brief introductory talk before a discussion.

However, if there are people willing to give a tutorial talk, we can also shorten or remove my talk from the BOS

[HeidiSeibold]

So what I need from you to be able to make a decision on this breakout are the following infos:

Who could be interested in collaborating on this?

Among others: @jakob-fritz @MakisH @led02 @pgierz @uekerman @chillenzer @rcaspart @trahflow

How much time do you need for this?

90 minutes should be ok, looks like we rather start with a get-together than a full-fledged event.

Abstract

Continuous integration techniques, or more general Cx techniques have become the de-facto standard (and synonym) for running various tasks whenever code changes in a repository. These tasks range from running a full-fledged test suite, creating new binaries/containers/documentations, to checking code styles. Yet, what are "good practices" in this field? Should you rather run/trust Github Actions or your own local Gitlab runner? Can you do both? How do you do Cx on HPC hardware? We would like to discuss these and other questions during this break-out session, which should bring together experts from various fields as well as interested attendees eager to learn what this is about. We plan to have a short introductory talk, followed by a discussion on the various questions and planning for future activities.

[jakob-fritz]

As I cannot edit, here my reply (partly doubles with edit from @pancetta)

Who could be interested in collaborating on this? From my feeling (and who provided a "thumbs-up") I assume at least the following people to be interested: @uekerman @pgierz @chillenzer @rcaspart @MakisH @led02 @pancetta Probably some more, when they see it in the schedule. I also talked with some people about the topic, but don't have their Github-Handle.

How much time do you need for this? 90 Minutes should be fine. It could be longer, but I assume, that it makes sense to keep it short as a starting-point

Abstract WIP

Would Tuesday Morning be doable for you? I will be there the whole time, so no constraints from me. But Tuesday Morning is no conference. I assume, you mean Thursday Morning? (Just for clarity. As said, I am happy with any time-slot)

[pancetta]

@jakob-fritz please check the abstract and ping me if you spot BS or want to add stuff.

[jakob-fritz]

@jakob-fritz please check the abstract and ping me if you spot BS or want to add stuff.

@pancetta Looks good to me. Thanks for writing

[MakisH]

So, it looks like we have plenty of items we could talk about. How about we have some opening talk, giving an introduction to the topic, and then we start to gather these items, make a wish list and decide how we could continue with this? Unless.. we actually find people willing to give short tutorial talk as proposed by @MakisH.

I meant that multiple people give short "pitch talks" about tools / features / concepts, and then they discuss them in small groups.

In that context, I could bring together a short talk on how (we try) to combine components from different repositories, if there is interest, as @uekerman mentioned above:

Something that is more and more important for us (meaning the research software preCICE): How to test over multiple repositories with downstream and upstream dependencies. @MakisH

We currently approach this with multiple Docker images and Docker Compose, but I am looking into different ideas at the moment.

[pancetta]

Hi! I just added the "Accepted" label to this BOS. Welcome on board! https://un-derse23.sciencesconf.org/program

[pancetta]

Hi all, the unconference is only 3 weeks away now! On day 1 there will be a breakout blitz where all session organizers should advertise their sessions. 1 minute, 1 slide, let people know what you intend to do. Please prepare this slide in advance and add it right here (PDF please), by September 20.

[uekerman]

I will unfortunately not be able to join the un-deRSE23 :disappointed:

[jakob-fritz]

Hi all, the unconference is only 3 weeks away now! On day 1 there will be a breakout blitz where all session organizers should advertise their sessions. 1 minute, 1 slide, let people know what you intend to do. Please prepare this slide in advance and add it right here (PDF please), by September 20.

Here the slide for the breakout blitz Breakout_blitz_CI.pdf

If someone thinks, the slide should be changed, please tell me with info on what needs changing, or propose a different slide.

[MakisH]

Thank you for preparing the slide, @jakob-fritz. It looks very nice! Should we prepare something else till then, or let it flow as it comes? I am happy to prepare something (collection of topics, or even some mini-talk), I just don't know what yet.

[pancetta]

Sure! If you have a topic for a pitch talk or want to show something, go ahead. The same is true for everyone else around here, of course 😃

[MakisH]

With around a week remaining, let me summarize a bit what we have so far.

Time: 90min -> This is probably too short for my previous suggestion of having mini-tutorials in groups. Let's indeed keep it at a "getting to know each other" level.

Potential speakers:

• @jakob-fritz: "An introductory talk before the discussion". @jakob-fritz what would you plan for this talk? Unless you mean the Blitz slide, which is already a very good appetizer.
• myself: "Testing the multi-component preCICE ecosystem". I propose this talk because I can touch many different tools and aspects in it. We have an upstream->downstream hierarchy of components, across different repositories, and we need to run regression tests whenever changing any of the involved components, while trying to save resources. My talk will involve GitHub Actions, Docker, fieldcompare, as well as a number of other CI tools used in different repositories. Problems involved include saving resources, running "proprietary" tests that cannot be published, and developing across multiple repositories. I can prepare a talk of 15-20min about this, I have some material and ideas already. Please upvote if interested, and please comment if you have specific questions already.

Topic ideas:

• @pancetta:
  • what it is, how it works, examples
  • Github Actions vs. Gitlab runners
  • how do you do CI for HPC codes?
  • what about continuous benchmarking?
  • continuous integration or continuous testing?
• @led02:
  • Providing dedicated / special executors (e.g., to use user-locked licensed software in the loop)
  • CI and hardware-in-the-loop testing
• From the abstract:
  • What are "good practices" in this field?
  • Should you rather run/trust Github Actions or your own local Gitlab runner?
  • Can you do both?
  • How do you do Cx on HPC hardware?
• myself:
  • Another aspect (maybe in parallel or after the talks) could be collaboratively creating a map of tools, with short description and common use cases. We could even compare/relate some of them (e.g., "Apptainer is like Docker, but you don't need root access").

My interests and goals

Of the above ideas, I am particularly interested in:

1. How to combine GitHub Actions and GitLab CI (e.g., for avoiding lock-in and for resilience)
2. How to do Cx on HPC hardware
3. Providing dedicated / special executors

If someone wants to give a talk on these topics, I would be most interested to attend it! :smiley:

I am a bit worried about the short time and the breadth of the topic. I think it would be useful to start with finding some common ground with questions of the kind "who already knows about X, Y, Z?" and focus on the new aspects. We could have a whiteboard / flipchart paper and vote about topics we would like to touch before the session starts.

I would enjoy finding out who has similar interests as I have and tries to solve similar problems.

I would also enjoy learning about the existence and basic concepts of useful tools, as well as forming a picture in my mind of what practices and tools are the most applied/used in our community, and what is still rather a desire than a common practice.

Suggested agenda

With the current state, I suggest starting with the suggested talk, to touch a range of topics on a concrete project (20min). @jakob-fritz could then take over the moderation of the discussion that this triggers, starting with the topics people have expressed interest in discussing (here or on an asynchronous on-site voting, to save time).

Since I understand that we are really self-organized, please upvote if all this makes sense, or please stop me if I have formed the wrong impression of what we want to do and what would be interesting.

[jngrad]

If you already have a HackMD or scratch pad for this BOS, I would be happy to put down some elements of answers based on my own experience with CI/CD.

• continuous integration and continuous benchmarking for HPC codes
  • ReFrame for testing and benchmarking
    • example: ReFrame software testsuite from the CSCS Swiss National Supercomputing Centre
    • example: ReFrame infrastructure at the National Energy Research Scientific Computing Center
  • JUBE Benchmarking Environment
    • example: JuPeX project and JUBE config files from the Jülich Supercomputing Centre
• Jupyter notebook continuous integration and delivery
  • testbook for unit testing
  • at the ESPResSo project, I maintain custom code to convert Jupyter notebooks to scripts to run integration tests with code substitution (e.g. to shorten simulation times or disable the 3D visualizer GUI), or to run notebooks and deploy them as static HTML files to GitHub pages (e.g. ferrofluid tutorial); the notebook themselves are not directly executable because multiple code fragments are written as markdown fenced code blocks which are hidden with the exercise2 plugin, thus a solution like testbook wasn't suitable
• how to combine GitHub Actions and GitLab CI
  • both services accept Docker images and have a Docker registry built-in; if Docker is an acceptable solution at your institute, you can easily generate a containerized environment that will behave the same way whether it's running inside a GitHub runner or a GitLab runner
  • if Docker is not an option in your infrastructure (due to the root privileges it requires), you can use GitHub and GitLab base images and install the necessary dependencies, then save them using a cache for faster load times, but the code that manages the cache doesn't use the same syntax in GitHub and GitLab
  • GitHub and GitLab can communicate through their API with a token, however maintaining the secrecy of the token is delicate if you want to allow everyone to open a PR or MR, and not just trusted members of the project

[jngrad]

A "RSE for HPC" working group was formed during deRSE23 at the initiative of @CaptainSifff. The main topics were about CI and benchmarking for HPC. You can find the meeting notes here. We haven't met again since April, in part because members of the group were quite busy working on the "RSE position paper no. 002" and the "Teaching RSE" papers.

If there is enough interest for CI in HPC, maybe we could revive the working group?

[MakisH]

If you already have a HackMD or scratch pad for this BOS, I would be happy to put down some elements of answers based on my own experience with CI/CD.

Let's start with this, we can clean it up later: https://pad.gwdg.de/s/Tzg1yx3ae#

If there is enough interest for CI in HPC, maybe we could revive the working group?

Sounds quite interesting and related! I am getting a bit overwhelmed of regular working group meetings at the moment, but I think a summary of your main findings in the group so far would be helpful. We can then see if people want to revive it (but April is not too long ago, to consider it gone).

[jakob-fritz]

My idea on an introduction was to give a brief talk (15 min) where I want to touch a few topics to spark discussion. Some of the topics that I planned to touch are the same, that I talked about in my presentation in the HiRSE Summer of Testing on Youtube.

But also some of the topics that you put in the list under @pancetta will be in the presentation. One of the topics will (briefly) be the combination of Github-Actions and Gitlab-CI.

After the introductory talks, I can also moderate the discussion.

[jngrad]

@MakisH an official pad has just be created for this breakout session; should the contents from your pad be transferred over to the newly created one, or should it be kept separate as a pre-session pad?

[MakisH]

@MakisH an official pad has just be created for this breakout session; should the contents from your pad be transferred over to the newly created one, or should it be kept separate as a pre-session pad?

Feel free to copy anything you find useful, in any format it best serves the session. I just wanted to create something to get the discussion going. :slightly_smiling_face:

[pancetta]

Here is the main hub for taking notes: https://pad.gwdg.de/FkFJTslFQhq-UF3Es6q4rw#

[pancetta]

Have fun with the session(s)! Please add the pad you're using also here for people to see what you did.

If possible, please prepare a 1 minute wrap up of your session for the farewell session on Thursday afternoon! What did you do in the session, how would you like to continue, how can people contribute after the unconference etc. We'll go through the blitz slides again one by one as in the blitz session.

[jakob-fritz]

A "RSE for HPC" working group was formed during deRSE23 at the initiative of @CaptainSifff. The main topics were about CI and benchmarking for HPC. You can find the meeting notes here. We haven't met again since April, in part because members of the group were quite busy working on the "RSE position paper no. 002" and the "Teaching RSE" papers. If there is enough interest for CI in HPC, maybe we could revive the working group?

Update on the working group mentioned above: The next meeting will take place on Oct 5th 15:00 CEST. We will meet here: https://webconf.fz-juelich.de/b/fri-4vb-wpv-tsk