search

openmc-dev / openmc

OpenMC Monte Carlo Code
https://docs.openmc.org
Other
772 stars 497 forks source link

Potential CI Service Transition #1731

Open pshriwise opened 3 years ago

pshriwise commented 3 years ago

@paulromano and I were discussing Travis' plans for changing its CI billing model.

https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

The new model doesn't require payment, seeing as this is still a public project, but it does make our build time much more restrictive (~10k minutes per month). Given that a conservative estimate for a single CI build for us is ~240 minutes, I don't think that's going to cut it. :grimacing:

I'm creating this issue to hopefully start a discussion on how to address this going forward. In the likely scenario where we move a new service, some of the alternatives brought up were:

I'm all ears for any thoughts or additions to the list!

pshriwise commented 3 years ago

The only one of these I have experience with is Circle CI, which makes for some really quick builds as it can cache images for specific steps in the CI build and really cut down on the run times. It can be a bit of a pain to set up though, and I'm not sure how well it will handle updating images for changes in our dependencies -- something I rather like about our current CI system. I know very little about GitHub actions (I believe @Shimwell has some experience there), but the idea of having CI in the same platform as the repo is attractive to me.

makeclean commented 3 years ago

When @shimwell see's this he'll be giddy :) There are a slew of benefits, one of them is automatic production of containers after merge, that was always true in Travis and Circle too, but I think it's easier with GH actions

From: Patrick Shriwise notifications@github.com Sent: Wednesday, November 25, 2020 11:32:21 PM To: openmc-dev/openmc openmc@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [openmc-dev/openmc] Potential CI Service Transition (#1731)

The only one of these I have experience with is Circle CI, which makes for some really quick builds as it can cache images for specific steps in the CI build and really cut down on the run times. It can be a bit of a pain to set up though, and I'm not sure how well it will handle updating images for changes in our dependencies -- something I rather like about our current CI system. I know very little about GitHub actions (I believe @Shimwellhttps://github.com/Shimwell has some experience there), but the idea of having CI in the same platform as the repo is attractive to me.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/openmc-dev/openmc/issues/1731#issuecomment-733990696, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AASTUSVBAEA677GPEREO273SRWHYLANCNFSM4UDAJI4Q.

shimwell commented 3 years ago

I have been very impressed with how easy Github Actions is to setup and how it has added features. The reviews comparing Circle and Github Actions I've read sometimes have updates as Github Actions has been improved even during the review. I would expect Github Actions to continue to add features and to have the benefit of having a single platform but the disadvantage of lock in to a particular platform.

This table offers straight forward comparison https://knapsackpro.com/ci_comparisons/circle-ci/vs/github-actions

I have some a minimal OpenMC test repositories that might be of interest for you https://github.com/Shimwell/example-git-actions-docker-openmc

We are actually trying to decided between the Circle and Github Actions over on the Paramak repo and are currently using both, we still can't decide even after using them both on the same repo:

The Github Actions community contributed templates help reduce the amount of code needed in the OpenMC repo to perform a particular opperation.

This 17 line file is all that is needed to build the docker, run pytest and upload the resutls to code coverage. https://github.com/ukaea/paramak/blob/main/.github/workflows/docker_ci.yml

Don't get me started on the actions for pypi uploading, auto pep8 and even automatic openmc benchmarking https://github.com/Shimwell/auto_openmc_benchmarking

pshriwise commented 3 years ago

Thanks for the info @Shimwell! It looks like a promising option.

shimwell commented 3 years ago

Just wanted to add that Github actions can be set to automatically run on a provided schedule using cron jobs.

  schedule:
    - cron: "0 0 * * *"

Thanks for sharing this cron command @RemDelaporteMathurin

So it could be set to run simulations every day and the resulting outputs can be extracted add plotted as svg graphs and then committed automatically to the repo along with some json or csv files of the results.

      - name: Commit files
        run: |
          git config --local user.email "action@github.com"
          git config --local user.name "GitHub Action"
          git add .
          git commit -m "Add new simulation results" -a
      - name: Push changes
        uses: ad-m/github-push-action@master
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          branch: ${{ github.ref }}

Combining this scheduled run feature and automated commit with some open source shielding benchmarks and I think we could have a nice continuous validation.

To get more compute power Github actions can also connect to the external runners if the simulations need scaling up.

Screenshot from 2020-12-09 17-40-35

paulromano commented 3 years ago

If I didn't know any better, I would have thought that you were getting paid by GitHub for all this free advertising you're giving them! :wink: GH Actions does seem to be a leading contender here.

In other news, our current CI is officially dead -- Travis stopped running our jobs because we are out of "credits". The fact that Travis never contacted open source maintainers to publicize this change and just left them out to dry does not sit well with me...

RemDelaporteMathurin commented 3 years ago

FWIW it seems that practically everything CircleCI does can be done in GH actions as @Shimwell said. The major difference I find is that CircleCI has a dashboard dedicated to testing. Makes the reports clearer in my opinion.

pshriwise commented 3 years ago

@Shimwell @paulromano and I chatted about this today. Seeing as the effects of the Travis update have now hit us

Builds have been temporarily disabled for public repositories due to a negative credit balance. Please go to the Plan page to replenish your credit balance or alter your Consume paid credits for OSS setting.

we're going to move forward with GitHub Actions so we can continue reviewing and merging PRs. I have a setup that replicates our Travis jobs that I'll hopefully be submitting later today.

pshriwise commented 3 years ago

Planned improvements for GHA CI:

shimwell commented 3 years ago

Just wanted to mention that I recently added some CSG and CAD based simulations of the same geometry to check that both methods produce similar results. This is done in a the CI so I thought it might be might be of interest https://github.com/ukaea/paramak/pull/642

drewejohnson commented 3 years ago

Just wanted to throw in my two cents that the GitHub actions platform looks excellent and works very well. I worry if/when we'll hit a limit for running these actions though. Not sure where we fall in the fee chart, but the free tier listed there has only 2000 minutes for actions

paulromano commented 3 years ago

That 2000 minute limit is for private repositories I believe. If you have a public repo, there is no limit as far as I can tell.

drewejohnson commented 3 years ago

Good to close?

paulromano commented 3 years ago

@pshriwise has a list of planned improvements in here, but perhaps we should make those separate issues?

pshriwise commented 3 years ago

Yeah, let's break those out into issues. I'll take care of that shortly. Thanks for reminding us about this @drewejohnson!