search

AllenInstitute / AllenSDK

code for reading and processing Allen Institute for Brain Science data
https://allensdk.readthedocs.io/en/latest/
Other
343 stars 149 forks source link

Consider caching dependencies for rc branch CI builds #1093

Open njmei opened 5 years ago

njmei commented 5 years ago

Describe the use case that is addressed by this feature. Currently our CIs spend the majority of their time installing dependencies from scratch. It would save a lot of time (I'm estimating 10-20 minutes saved per CI build) if these dependencies were cached so that the CI could get to tests much faster.

Describe the solution you'd like If we could implement caching of dependencies for CI builds of PRs merging into rc (release candidate) branches this could greatly speed up iteration speed. We would still want to have a full 'from scratch' build when merging rc into master to be thorough.

NileGraddis commented 5 years ago

@njmei good idea!

questions:

  1. Is this feature supported by our CI providers or do we have to roll our own using generic persistent storage? I am pretty confident that it is the latter given that our config files seem to have a bunch of custom install commands, but maybe that is all unnecessary?
  2. How are we going to handle changes which update the requirements?
  3. How are we going to detect when an unpinned dependency updates and breaks us?
njmei commented 5 years ago

  1. Travis supports caching: https://blog.travis-ci.com/2016-05-03-caches-are-coming-to-everyone Appveyor support caching: https://www.appveyor.com/docs/build-cache/ CircleCI support nicer caching: https://circleci.com/docs/2.0/caching/

  2. We will probably need to write some custom logic for Travis/Appveyor that is similar to: https://circleci.com/docs/2.0/caching/#using-keys-and-templates. But it's possible that such functionality already exists and I just didn't look hard enough.

  3. We should be able to set up CI configurations so that caches are not used for rc -> master PRs. Unpinned dependency update breakages would be caught in the rc -> master PR.