Make test suite more efficient to run in 10 minutes

weiji14 commented 4 years ago

Description of the desired feature

PyGMT's test suite is growing, and in the long-term, we'll need to make testing a bit more efficient for contributors. As a goal, we should aim for ~10 minutes for the whole test suite (Windows takes up ~\~30min~ 15min now (see #586)), and <1 minute for local tests on just one module.

Here's a few ways we can tackle this (note to maintainers, feel free to edit this top post):

Local tests

Running make test can be slow since it runs everything, but sometimes we only want to test one function (e.g. grdimage).

[x] E.g. at https://github.com/GenericMappingTools/pygmt/pull/516#discussion_r464184518, we should document that contributors can run just one test using something like pytest -v --mpl pygmt/tests/test_grdimage.py.
[ ] Alternatively, look into something like pytest-fastest to automate testing only the code that has changed between the current branch and master.

Github Actions tests

Installing dependencies

The bulk of the CI time is spent installing dependencies from conda (4min on Linux, 9min on Windows). We could:

[x] Cache dependencies. This was experimented before, but it didn't seem to help (see https://github.com/GenericMappingTools/pygmt/pull/475#issuecomment-656392670), could try again to see things have changed
[x] Try switching to mamba which is a fast reimplementation of conda written in C++. I've tried it before, but it's still in experimental mode at setup-miniconda, see https://github.com/conda-incubator/setup-miniconda#example-6-mamba (#841)

Speed up full test suite

Find a way to run a minimal set of tests when Pull Request is in draft, and only full test suite when Pull Request is ready to review/in review?

Are you willing to help implement and maintain this feature? Let's do it together :smile:

willschlitzer commented 3 years ago

Find a way to run a minimal set of tests when Pull Request is in draft, and only full test suite when Pull Request is ready to review/in review?

@weiji14 Would there be a way to just have the test suite run on one machine (presumably Ubuntu, since that typically is completed in the shortest period of time) in one version of Python when the PR is a draft and then run the full suite when it is ready for review?

Also, would reducing the test suite have any impact on how long it takes for the vercel preview deployment, or is that handled separately?

weiji14 commented 3 years ago

@weiji14 Would there be a way to just have the test suite run on one machine (presumably Ubuntu, since that typically is completed in the shortest period of time) in one version of Python when the PR is a draft and then run the full suite when it is ready for review?

Yes it's possible, but you'll need a bit of Github Actions CI experience. Have a look at the GMT Latest tests .yaml file for inspiration. And yes, running just Ubuntu Python 3.9 in draft mode is the idea.

Also, would reducing the test suite have any impact on how long it takes for the vercel preview deployment, or is that handled separately?

Vercel documentation build is separate, speeding that up would require more work, either from optimizing our Gallery/Tutorial examples or some form of caching on the Python/Sphinx or Javascript/Vercel side. But you could find a way to measure/benchmark it first to see where things run slowest.

willschlitzer commented 3 years ago

@weiji14 I'm trying to make a draft test suite, but I'm getting my branch remote rejected because of the OAuth app. It looks like I should be able to generate a personal access token in Developer Settings, but PyGMT isn't listed as an application when I go there to get a token. Any idea how I should go about fixing this?

weiji14 commented 3 years ago

Try setting your remote using ssh (git@github.com:GenericMappingTools/pygmt.git) instead of https (https://github.com/GenericMappingTools/pygmt.git) and see if it works.

willschlitzer commented 3 years ago

I was able to change the remote but received:

git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

weiji14 commented 3 years ago

You also need to add your ssh key to your GitHub account https://docs.github.com/en/github/authenticating-to-github/adding-a-new-ssh-key-to-your-github-account.

willschlitzer commented 3 years ago

It works! Thanks @weiji14!

seisman commented 7 months ago

I think we can close the issue since we already achieved our initial goal by a series of PRs. Now the Tests workflow takes 2 minutes on Linux and <8 minutes on Windows.

[ ] Alternatively, look into something like pytest-fastest to automate testing only the code that has changed between the current branch and master.

I think we won't use pytest-fastest, mainly because:

It's still in very early development with a beta version v0.0.12
It's not actively maintained
It doesn't support Python 3.12 yet

Instead, we may use a similar alternative tracked in #2791.

Closing the issue now.

GenericMappingTools / pygmt