Open GuillaumeBroggi opened 2 months ago
Thanks! This MemoryError
comes from numpy.random_generator.Generator.choice
.
We might think about an upper limit that is displayed in the documentation or find a work-around
If I am not mistaken, the incriminated lines are: https://github.com/bessagroup/f3dasm/blob/6b0da8b70196515f0153b05b9c7102bc23df3bda/src/f3dasm/_src/experimentdata/samplers.py#L180-L184
Indeed, the range
generator is evaluated for the choice
.
The issue with displaying an upper limit in the documentation is that it's a user-specific limit, depending on the hardware.
A workaround should avoid this evaluation. For an int
(inspired from https://stackoverflow.com/a/54082200, no extensive testing done):
import numpy as np
def current_implementation(n_samples, lower_bound, upper_bound, step):
rng = np.random.default_rng(seed=42)
return rng.choice(
range(lower_bound, upper_bound + 1, step),
size=n_samples,
)
def proposed_workaround(n_samples, lower_bound, upper_bound, step):
rng = np.random.default_rng(seed=42)
return (
lower_bound
+ rng.integers(
low=0, high=(upper_bound - lower_bound) / step + 1, size=n_samples
)
* step
)
current_implementation(n_samples=10, lower_bound=32, upper_bound=42, step=2)
array([32, 40, 38, 36, 36, 42, 32, 40, 34, 32])
proposed_workaround(n_samples=10, lower_bound=32, upper_bound=42, step=2)
array([32, 40, 38, 36, 36, 42, 32, 40, 34, 32])
current_implementation(n_samples=10, lower_bound=0, upper_bound=int(1e12), step=1)
MemoryError ...
proposed_workaround(n_samples=10, lower_bound=0, upper_bound=int(1e12), step=1)
array([773956048556, 438878439752, 858597919912, 697368029060, 94177347887, 975622351637, 761139701991, 786064305277, 128113632675, 450385937896])
The following code with a higher bound of 1e12 raises a
MemoryError
(the exact admissible value will depend on the machine).The error could be explained if the full int sequence is evaluated at some point (I did not inspect the code).