Closed tovrstra closed 7 years ago
The pip cache is only for the downloads I think. I'm not sure it'll cache the install itself (it would be better if it didn't, since it's likely to interact with other packages).
I'm currently just caching all of the miniconda directory. I'll remove the offending directories in the before-cache section.
Pip cache is only for downloads: https://pip.pypa.io/en/latest/reference/pip_install/#caching
Okay, caching is implemented! It sped things up from 4min->3min in the python-cython-ci-example. Not great, but it's something I guess.
Good. I'll take a quick look.
I think the speedup is purely random. In the older runs, the download and install of miniconda just takes a few seconds. Just compare the following two and look at the variation in timing of the steps that remained the same:
With such variability, there is no point in profiling (and caching) I'm afraid.
Hmm is that the case? I thought there's a 50 second reduction in the time to install the conda packages in the install.5 and install.6 sections?
I'm not sure how the total comes up to 3 minutes though on the cached job. A quick addition on https://travis-ci.org/theochem/python-cython-ci-example/jobs/270785980 doesn't even come up to 1 minute. Are there parts which are not being profiled properly?
On Fri, 1 Sep 2017 at 13:41 Toon Verstraelen notifications@github.com wrote:
I think the speedup is purely random. In the older runs, the download and install of miniconda just takes a few seconds. Just compare the following two and look at the variation in timing of the steps that remained the same:
- https://travis-ci.org/theochem/python-cython-ci-example/jobs/270785980
https://travis-ci.org/theochem/python-cython-ci-example/jobs/269529284
With such variability, there is no point in profiling (and caching) I'm afraid.
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/theochem/qcgrids/issues/8#issuecomment-326559531, or mute the thread https://github.com/notifications/unsubscribe-auth/AA_-NZWL0zzNR1APS0ajqkl30ub3PSKYks5sd-15gaJpZM4PI3d2 .
-- Matt
Sent from my phone
I'd guess CPU time versus wall time. Not sure.
Can we just postpone this for a while? This is not such an issue and it quickly clobbers the CI scripts.
Sure. We can work on splitting the builds based on tags first.
On Fri, 1 Sep 2017 at 14:48 Toon Verstraelen notifications@github.com wrote:
Can we just postpone this for a while? This is not such an issue and it quickly clobbers the CI scripts.
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/theochem/qcgrids/issues/8#issuecomment-326571472, or mute the thread https://github.com/notifications/unsubscribe-auth/AA_-NXMlzVMTyn5fMtGq7o60Nc_L-7VIks5sd_0ugaJpZM4PI3d2 .
-- Matt
Sent from my phone
@matt-chan It seems that caching is fully working! So we can close this issue I guess.
There is just one small thing: the .travis.yml
script is not self-contained at the moment due to one of the caching steps. (This is a source of mistakes when copying over .travis.yml
files to other projects.)
Okay, #13 should fix this?
I'm going to leave the issue open a bit more because I want to make sure our cache isn't being invalidated by spurious changes.
The current .travis.yml
installs pip packages into the conda env, which gets then cached. This is not ideal because they will not get update once cached. We could do pip install --user --upgrade ...
to avoid this issue. Pip caching can be made efficient as shown here: https://github.com/nickstenning/travis-pip-cache
That still has the disadvantage that wheels may accumulate over time. We have a wheel cleaning script in the HORTON repo to get rid of them: https://github.com/theochem/horton/blob/master/tools/qa/remove_old_wheels.py
It doesn't need to be in --user. I think running it with --upgrade should be enough. The miniconda cache will still be updated if there's a new package
On Sun, 3 Sep 2017 at 01:00 Toon Verstraelen notifications@github.com wrote:
The current .travis.yml installs pip packages into the conda env, which gets then cached. This is not ideal because they will not get update once cached. We could do pip install --user --upgrade ... to avoid this issue. Pip caching can be made efficient as shown here: https://github.com/nickstenning/travis-pip-cache
That still has the disadvantage that wheels may accumulate over time. We have a wheel cleaning script in the HORTON repo to get rid of them: https://github.com/theochem/horton/blob/master/tools/qa/remove_old_wheels.py
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/theochem/qcgrids/issues/8#issuecomment-326773843, or mute the thread https://github.com/notifications/unsubscribe-auth/AA_-NeJLrQHQafE3OUqxr652I4D1g0Amks5sed4ZgaJpZM4PI3d2 .
-- Matt
Sent from my phone
This is fixed.
I'm interested in speeding up our Travis builds that use conda environments. We specify our environment using an environment.yml
, which is currently reinstalled every build. The part of the install process that takes the longest is Solving package specifications
triggered by conda env create
.
I just wanted to confirm that your caching method discussed above alleviates this "Solving package specifications" step (it seems like it)? Also if so, is https://github.com/theochem/qcgrids/commit/157318c293ac05312ddc4864ad669710903c7712 the crucial commit to enable conda caching on Travis? Sorry if this is a bit off topic... this is the most relevant issue I could find!
Hi Daniel,
I'd actually take the Travis.yml on master, since the relevant changes were split over several commits.
The lines of interest are: 46-54 and 61-85.
The caveat is that you must manually ensure your conda environment is consistent. If you remove anything from your meta(environment).yml, you must invalidate the cache on Travis, or you must uninstall it in the Travis yml. Adding packages and updating them is unaffected.
The before_cache lines will make sure you aren't rebuilding your cache (takes about a minute or two) unnecessarily (those directories change when you do a conda build).
Take care! Matt -- Matt
Sent from my phone
Thanks @matt-chan!
The lines of interest are: 46-54 and 61-85.
For posterity / convenience, I've pasted those snippets below:
If you remove anything from your meta(environment).yml, you must invalidate the cache on Travis, or you must uninstall it in the Travis yml.
I wonder if there's a way to see if the changeset that's being built modified environment.yml
, and if so wipe the cache. Referencing https://github.com/nest/nest-simulator/pull/75.
On Fri, Oct 13, 2017 at 4:56 PM Daniel Himmelstein notifications@github.com wrote:
I wonder if there's a way to see if the changeset that's being built modified environment.yml, and if so wipe the cache.
You can detect changes in a file easily in a PR on Travis. Travis works on a merge of the PR and the branch being merged into. The error code of the following may be used to detect a change in any given file, here environment.yml
git diff $TRAVIS_BRANCH --stat environment.yml
I'm not sure what would be the best way to wipe the cache inside a .travis.yml file. You can obviously do a manual rm.
The following should be safe/needed for caching:
miniconda/LICENSE.txt
miniconda/bin
miniconda/conda-meta
(meta info on installed packages)miniconda/envs
miniconda/etc
miniconda/gcc
(not sure why it exists)miniconda/include
miniconda/lib
miniconda/lib64
miniconda/libexec
miniconda/sbin
miniconda/share
miniconda/ssl
Not good for caching:
miniconda/conda-bld
miniconda/locks
miniconda/pkgs
miniconda/var
There may be others.
This may not be a good idea because of things not to cache. See also broadinstitute/viral-ngs#290
This is how to enable it: https://docs.travis-ci.com/user/caching/#Arbitrary-directories
Other things to consider for caching in Travis: