Testing workflows - Githubissues

spacetelescope / style-guides

An opinionated guide on how we work.

Creative Commons Attribution 4.0 International

56 stars 33 forks source link

Testing workflows #7

Closed stscicrawford closed 5 years ago

stscicrawford commented 6 years ago

Add information about testing workflows.

stscicrawford commented 6 years ago

Some example:

@stsci-hack https://innerspace.stsci.edu/display/SSR/Adding+New+Tests%3A+A+User%27s+Guide

stscicrawford commented 6 years ago

Comments about testing workflows from #9

Also maybe mention something like "use urllib/astroquery/astropy.utils.data, not roll-your-own-downloader when remote data is required"?

One step further might even be "and use some pytest markers to indicate that these require downloading" - the astropy machinery for this is a lot easier to integrate now that it is a stand-alone and does not require the helpers.

pllim commented 6 years ago

cc @bhilbert4

pllim commented 6 years ago

Testing can cover several things:

Unit tests
Regression tests (big data, cross-testing, etc)
PEP 8 (Python)
Doc build (e.g., Python + Sphinx)
Coverage

pllim commented 6 years ago

Also mention different CI services to choose from:

Travis CI: Lightweight tests that don't exceed quotas for size and timing. Available for testing under Linux or OSX. Also see https://blog.travis-ci.com/2018-05-02-open-source-projects-on-travis-ci-com-with-github-apps (i.e., using travis-ci.com vs travis-ci.org ).
Jenkins: In-house support. Virtually unlimited (unless you are very crazy). Has access to Artifactory. Linux only.
Appveyor: Has quota like Travis. Running tests on Windows.
CircleCI, etc.

sean-lockwood commented 6 years ago

@pllim I might not understand the scope of this guide, but isn't the Institute trying to limit CI tools to Jenkins (for internal projects) and Travis CI (for outside collaborative projects)?

pllim commented 6 years ago

And links to pytest resources.And some plugins that might be useful for pytest:

pllim commented 6 years ago

Re: limit CI tools -- but does that mean we should never test under Windows? :woman_shrugging: (this is when I slowly back away towards the door)

pllim commented 6 years ago

Also guideline for when to bundle test data with the package, and when to use "big data" solution?

stscicrawford commented 5 years ago

I've assigned @pllim to take a first draft. After that is available, we can then iterate on it and add more material

pllim commented 5 years ago

@drdavella , want to be co-author? 😉

drdavella commented 5 years ago

Sure, I'm happy to help and/or review.

pllim commented 5 years ago

@stscicrawford , is there a deadline for this...? :grimacing: