Fixes incorrect selection of water molecule in Targeted Insertion

[badisa]

• Was sampling effectively randomly due to https://github.com/proteneer/timemachine/blob/fec45823b3c7990526f0cf0f8369de14062d96c2/timemachine/cpp/src/tibd_exchange_move.cu#L228-L236 using the this->d_log_weights_before_ rather than d_src_log_weights_.
• Was correctly selecting a water from the the right region thanks to https://github.com/proteneer/timemachine/blob/fec45823b3c7990526f0cf0f8369de14062d96c2/timemachine/cpp/src/kernels/k_exchange.cu#L791-L808 correctly pulling a water from the target region, but the probabilities could be much lower than you would want.
• Broken as part of https://github.com/proteneer/timemachine/pull/1235

Analysis

Previous analysis (where the code was incorrect) we had determined that the results were correct at about 50-50 occupancy of 1 or 2 waters in the buckyball. Two runs from 0 and 6 starting waters produces a different result in this case compared to the previous values we saw in https://github.com/proteneer/timemachine/pull/1228. Though in our HREX reference we have seen up to 70%-30% split, so its unclear if this is a convergence issue, a correctness issue or something else.

Used the following commands to generate the data

python -u examples/water_sampling_mc.py --water_pdb timemachine/datasets/water_exchange/bb_0_waters.pdb --ligand_sdf timemachine/datasets/water_exchange/bb_centered_espaloma.sdf --out_cif traj_0_waters.cif --md_steps_per_batch 10000 --mc_steps_per_batch 10000 --insertion_type targeted --use_hmr 1 --iterations 20000
python -u examples/water_sampling_mc.py --water_pdb timemachine/datasets/water_exchange/bb_6_waters.pdb --ligand_sdf timemachine/datasets/water_exchange/bb_centered_espaloma.sdf --out_cif traj_6_waters.cif --md_steps_per_batch 10000 --mc_steps_per_batch 10000 --insertion_type targeted --use_hmr 1 --iterations 20000

Starting from 0 Total MC Moves 200000000 Num Waters in Buckyball: 1, 57.50% Num Waters in Buckyball: 2, 42.49%

Starting from 6 Total MC Moves 200000000 Num Waters in Buckyball: 5, 0.01% Num Waters in Buckyball: 3, 0.03% Num Waters in Buckyball: 2, 34.73% Num Waters in Buckyball: 1, 65.24%

[badisa]

Offline @proteneer had asked for a plot with the rolling average

In this view it looks like they may both be converging to 1.5 (or 50% for each) over time

From the same data as the plots in the descriptions.

Another request was looking at the neutral buckyball case:

python -u examples/water_sampling_mc.py --water_pdb timemachine/datasets/water_exchange/bb_0_waters.pdb --ligand_sdf timemachine/datasets/water_exchange/bb_centered_neutral.sdf --out_cif traj_0_waters.cif --md_steps_per_batch 10000 --mc_steps_per_batch 10000 --insertion_type targeted --use_hmr 1

Still preliminary since it hasn't been running long:

Waters are accepted for a short time then rejected almost immediately

Significantly more accepted proposals compared to the charged case

Charged case for reference: