Some modifications to prevent the (40min+?) time it previously took to plot frequency distributions. I also noticed that it consumed all my 64G memory and I wasn't able to complete it locally anyway, so there are some memory optimizations too.
only reading the columns we need in pd.read_csv.
discarding memory held by variables (lists, series etc) we don't need anymore, by reusing variable names
.tolist() at a couple of places where lists will suffice, to avoid the memory overhead of pd.Series
generation of pd.DataFrame using a list of dicts instead (supposed to be the fastest way to construct a DataFrame).
Not using sns.boxplot directly but in a slightly roundabout way (this was the main culprit in the long runtime).
logging at several steps to avoid wondering what's going on.
Also removed some configuration options from Snakefile_model_eval which are not used (seeds is gone in favor of train_seeds and sample_seeds, for example, so perhaps its best not to get config variables we don't actually use).
Some modifications to prevent the (40min+?) time it previously took to plot frequency distributions. I also noticed that it consumed all my 64G memory and I wasn't able to complete it locally anyway, so there are some memory optimizations too.
pd.read_csv..tolist()at a couple of places where lists will suffice, to avoid the memory overhead ofpd.Seriespd.DataFrameusing a list of dicts instead (supposed to be the fastest way to construct a DataFrame).sns.boxplotdirectly but in a slightly roundabout way (this was the main culprit in the long runtime).logging at several steps to avoid wondering what's going on.
Also removed some configuration options from
Snakefile_model_evalwhich are not used (seedsis gone in favor oftrain_seedsandsample_seeds, for example, so perhaps its best not to get config variables we don't actually use).