Avoid repeatedly instantiating potentials

[badisa]

• Re-using the preallocated potentials appears to get us back some performance

Benchmarks

The reference refers to what is currently on master, the comparison is this branch. Generated using the test here: https://github.com/proteneer/timemachine/blob/1b8f07f102cdeab76f5b7e2126dcd8ba7bfcc49e/tests/test_benchmark_free_energy.py#L76 with the additional change to run with 2, 4, 8, 16, 32 and 48 windows as well as making sure every window always ran 100 frames.

Run on an A10 GPU with Cuda Arch 8.6. Only sampled once, but think the differences are significant even without replicates.

Also ran a small graph (bisection + hrex + water sampling) that was about 5% faster with this change.

Vacuum

Doesn't make a difference on vacuum simulations, presumably the cost of constructing potentials is cheap (no neighborlist notably, tiny systems, etc)

Solvent

Seems to make the biggest difference for bisection

Complex

Without water sampling

With water sampling

[badisa]

Would you mind adding these changes (+ either the plot or some other convenient export of the resulting timings) to test_benchmark_free_energy.py ? Will be good to track timings across this grid of conditions in the future.

Added in https://github.com/proteneer/timemachine/pull/1287/commits/19b5c7d655e32c6ad2b3492717f0021dc9591298

Didn't change the tests as the settings can take several hours to run and doesn't seem worth it to add to the nightly tests for just timings. This now more closely mirrors our logic around the other benchmarks: https://github.com/proteneer/timemachine/blob/9c023081c661ec320d2198d8f766257b5d13debf/tests/test_benchmark.py#L524-L533

[badisa]

Out of curiosity I ran vacuum and solvent over night going out to 2000 frames to see if the performance differences changed. Only noticeable difference was that with solvent HREX only ~20%~ 30% of ns/day was lost versus 65% lost in vacuum. Which makes sense as solvent is more likely to be dominated by the cost of MD rather than by the cost of whatever work these methods do on their own.