HEmile / storchastic

Stochastic Automatic Differentiation library for PyTorch.
GNU General Public License v3.0
180 stars 5 forks source link

Automatically join samples whenever they are duplicates #66

Open HEmile opened 4 years ago

HEmile commented 4 years ago

Monte Carlo methods for discrete RVs with replacement should not waste computation on duplicate samples if the output dynamic is deterministic. We could automatically join these samples in this way:

As deterministic calls are/should be deterministic, we can guarantee this doesn't provide incorrect behaviour. Downside: If indexing is slower than the function performance, this could actually be much slower if a lot of deterministic calls happen...

Another option is to have the user select whenever this joining should happen, using eg a context wrapper.

HEmile commented 4 years ago

Other issues: If there are more plates, the amount of unique samples can vary, meaning we get empty tensors that are still in use. Taken as a single tensor, if there is a single element in the plate for which there are no duplicates, then we still compute it for the whole sample performance.

Therefore, to make proper use out of it, this would require looping over the function for individual dimensions, otherwise you still do the full computation... Not sure if it's worth it.