search

InfluenceFunctional / MXtalTools

BSD 3-Clause "New" or "Revised" License
7 stars 1 forks source link

Reconfirm/streamline RAM usage #24

Closed InfluenceFunctional closed 1 year ago

InfluenceFunctional commented 1 year ago

During big runs on cluster, we get infrequent RAM spikes up to 40 GB. I'm assuming during reporting but not sure.

In any case, find the bottleneck and ease it - asking for 40GB that we mostly don't use is quite wasteful.

InfluenceFunctional commented 1 year ago

Appears so far that it's in the initial dataset loading.

InfluenceFunctional commented 1 year ago

Attempted to smooth it out to limited success.

Perhaps need to load and process the dataset piecewise so that we don't have two or three huge duplicates just sitting around.

InfluenceFunctional commented 1 year ago

complimentary to #63