AGBV / YASF

Yet Another Scattering Framework python implementation
https://agbv.github.io/YASF/
MIT License
0 stars 0 forks source link

Splitting data into data batches #24

Open arunoruto opened 9 months ago

arunoruto commented 9 months ago

Currently, the particle_interaction function implemented on GPU is taking the whole dataset. This can lead to out of memory errors, especially on GPUs with less VRAM. To mitigate this problem, the data should be batched into smaller chunks before processing!

This should be addressed by calculating the needed and available memory on the GPU. Dividing these two values gives a lower bound on the number of batches to divide the dataset into.

arunoruto commented 8 months ago

Two approaches:

aetherspritee commented 7 months ago

check #38

arunoruto commented 7 months ago

Currently, only the phase function is batched due to the large number of angles that sometimes require calculation. The approach should be generalized further for other functions.