perform tests with cold and warm cache

[dimbleby]

add cold/warm cache variants to "lock" and "update". Sample results here, via the gh-pages branch of my fork.

Various things one could quibble with here...

It's a bit unclear how "cold" the cold versions should be:

• does it make sense to remove the lockfile before "lock warm"?
  • I think yes, it feels to me as though the point of "lock warm" is to create the lockfile.
  • Probably it doesn't make a difference
• why not remove the virtualenv before "update cold"?
  • the answer is that I tried that but one of the tools (I forget which!) expects that the virtual environment is there in its interpretation of "update", so it just doesn't work
  • well all benchmarks are flawed, clearly "update" does not mean the same thing for the different tools, what can you do?

I would have done cold and warm for "add package" too but it's a little harder: because after you've added the package the first time, it's not quite trivial to restore the state in readiness for adding it again.

I haven't thought too hard about what the website will look like during the 24 hours where some workflows are the old ones and some are the new ones. I believe the result should be three bars on some of the charts, which would seem tolerable - but that is only a thought experiment.

[ipmb]

Thanks for the PR. Sorry it took me so long to get to it.

I'm open to redefining or clarifying what these mean if you see a place that it can obviously be improved.

[dimbleby]

thanks!

one thing that this does reveal is that the CI pipeline is configured such that poetry's virtual environment is inside poetry's cache. So for poetry, cleaning the cache is also cleaning the venv.

in particular, the cold version of poetry-update is paying the penalty for this: instead of updating the dependencies and seeing that nothing has changed, it has to update the dependencies and recreate the virtual environment and install them all.

Arguably this is all fair game - it was poetry's choice to put the venv in its cache by default - but arguably it would be a more even comparison if the benchmark configured poetry to put its venv somewhere else.