Use Azure pipelines for tests

minrk commented 5 years ago

repo2docker tests take a long time! There's not a whole lot we can do about this other than try to increase parallelism and caching (these are usually mutually exclusive) and use more unittests instead of full builds. We're already well passed the limit of parallelism available on Travis.

One way to increase parallelism is to switch to a CI provider other than Travis that offers more at the free tier. It's been popular recently to use Microsoft's Azure Pipelines for CI for projects that have really high demand (conda-forge, jupyterlab) and I think they offer the most for free (presumably since they are a. Microsoft and b. late to the game). Another, perhaps more tedious option, would be to spread tests over a few CI services if one service doesn't offer the parallelism we need, but that might make the maintenance burden more than it's worth.

cc @SylvainCorlay who brought up r2d tests clogging the Jupyter account travis queue.

betatim commented 5 years ago

At EuroPython I chatted with @ogrisel about CI and he said he was a happy camper with Azure Pipelines and found that (because they were late to the party) the queues are short and things run fast.

I think we should try out moving to a different provider and only if that doesn't work out consider splitting because of the additional maintenance you mentioned. Going to a different provider would also fix the pain we cause others (as long as they don't also switch ;) )

trallard commented 5 years ago

I recently helped Carol et al. to migrate to Azure pipelines for a couple of repos to in the nteract org. Thus I'd bee happy to implement this in a fork of mine of repo2docker so that you can compare?

betatim commented 5 years ago

I'd bee happy to implement this in a fork of mine of repo2docker so that you can compare?

Could we add it to the main repo and then run things in parallel? That way we'd have the two right next to each other in each PR which makes it easy to build up a feeling for which one is faster, more stable, nicer output, $stuff. Do you need an admin to enable/disable things in the settings?

trallard commented 5 years ago

Yes- I was suggesting implementing on a fork of mine as I would need elevated privileges to enable the Azure DevOps Github app

betatim commented 5 years ago

@minrk @Carreau or @willingc could one of you add the https://github.com/marketplace/azure-pipelines app to this repository please? (I think that is the right app to add)

trallard commented 5 years ago

I can work on this send the PR and on merge, this can be added/approved by @willingc (that's what we did for scrapbook - of course if Carol is happy with this)

willingc commented 5 years ago

Will do in a bit..

betatim commented 5 years ago

Thanks to @willingc and friends we now have a basic pipeline on Azure 🎉!

Things to do to get it up to "full production readiness" aka "good enough to replace travis":

[ ] is there a way to run the unit and lint jobs first (they complete quickly so give feedback early on)
[ ] upload coverage results to codecov, see this blog for ideas
[ ] run the "venv" (and "unit") tests with Python 3.5
[ ] a copy of the deploy setup we have that creates a release on PyPI when we create a new tag
[ ] notifications only for failed builds on push to master

(if you remember more things please mention them)

willingc commented 5 years ago

A big thank you to @trallard for helping me set Azure Pipelines on another repo and for teaching me the general process 💐

betatim commented 5 years ago

I made the things I thought of a checklist so people can pick one (or many) and we can tick them off one by one.

I started receiving emails about successful builds on Azure which is a bit too noisy for me. I disabled notifications for me personally, I couldn't find how to set it on a team level via the azure-pipelines.yaml. Even better would be if we could only have notifications for broken builds on master.

nuest commented 5 years ago

It seems the Azure pipelines do not propagate the test failures:

https://dev.azure.com/jupyter/repo2docker/_build/results?buildId=25&view=logs&jobId=7ff9283a-ab30-5e9f-8967-b9fdc546360c

ends with

================== 6 failed, 168 passed in 758.08s (0:12:38) ===================
~/work/1/s
##[section]Finishing: Run tests unit

But the job is still "succeeded".

betatim commented 5 years ago

This is really weird. I am 99% sure pytest exists with a status code != 0 when something goes wrong. And this test https://dev.azure.com/jupyter/9d4b17b6-9c19-4575-915f-3fefb185309a/_build/results?buildId=30&view=logs&jobId=1275e120-8bca-5e65-aa95-8702b87a4eab got marked as failed because the script exited with status code 2. So Azure pipelines will fail when the exit code is not zero.

manics commented 5 years ago

https://github.com/jupyter/repo2docker/blob/371b925adf02bfa4bbe51995fb5051579053d140/azure-pipelines.yml#L65

Perhaps you've run into this: https://github.com/microsoft/azure-pipelines-yaml/issues/135

manics commented 4 years ago

We've switched to GitHub actions/workflows: https://github.com/jupyterhub/repo2docker/pull/942

jupyterhub / repo2docker

Use Azure pipelines for tests #744