The data preparation cascade and ccf.xcor() are memory-intensive because they copy the data. This causes trouble when running on an ordinary laptop (16GB RAM) and ESPRESSO data, which is natively very large.
The data is read in as a list_of_orders and an equally sized list_of_errors (uncertainties) (Now the data is in memory twice).
Various data cleaning steps are then performed, overwriting these two lists.
In anticipation of model injection, the lists are copied when doing normalization; leaving list_of_orders (to be used later for model injection) and list_of_orders_normalised (Now the data is in memory four times).
Then during xcor, the normalised data and errors are copied once again when calling np.hstack (Now the data is in memory 6 times).
Memory usage then explodes in xcor() when casting the template in a form suitable for matrix-multiplication, because the template is then interpolated onto the data in an array that measures n_wl x n_RV. As n_RV may be ~1000; this is equivalent to an instance of the data with 1000 exposures. Typically this would be equivalent to 20 datasets if my dataset has 50 exposures each.
In total, this would amount to needing to place the data into memory 6+some 20 times. For a 1.6GB ESPRESSO dataset; this is 40 GB.
Unsurprisingly, when running this on my 16GB RAM macbook, by the time xcor() is being executed, it is using 20 GB in swap memory. Laptop then almost freezes (though seems to grind on). But as I rarely keep more than 20 GB in free disk space, this is a disaster zone.
How to address this?
1 Implement a check that deletes list_of_orders from memory after normalization, if inject_model is not called; saving 2 instances of the data in memory before.
2 Write list_of_orders and list_of_errors to drive in xcor(), then delete them from memory after hstack (sigh...), OR provide h_stacked orders to xcor to begin with (sigh...). I don't like either of these, but these would save another 2 instances of the data in memory.
These two, sadly, are peanuts compared to the casting of the template... I could implement point #1 as it is trivial to implement; but then also add a conserve-memory functionality in start_run and xcor(); where the multiplication of the template is perfomed the slow, for-loopy way; or perhaps in chunks of RV. Highly annoying.
For now, the cheapest way to conserve memory is to lower the number of RV-steps; but that limits the CCF filter-width...
Of course, this problem only occurs for template-based correlation. Template masks are tiny in memory.
The data preparation cascade and ccf.xcor() are memory-intensive because they copy the data. This causes trouble when running on an ordinary laptop (16GB RAM) and ESPRESSO data, which is natively very large.
The data is read in as a list_of_orders and an equally sized list_of_errors (uncertainties) (Now the data is in memory twice). Various data cleaning steps are then performed, overwriting these two lists.
In anticipation of model injection, the lists are copied when doing normalization; leaving list_of_orders (to be used later for model injection) and list_of_orders_normalised (Now the data is in memory four times).
Then during xcor, the normalised data and errors are copied once again when calling np.hstack (Now the data is in memory 6 times).
Memory usage then explodes in xcor() when casting the template in a form suitable for matrix-multiplication, because the template is then interpolated onto the data in an array that measures n_wl x n_RV. As n_RV may be ~1000; this is equivalent to an instance of the data with 1000 exposures. Typically this would be equivalent to 20 datasets if my dataset has 50 exposures each.
In total, this would amount to needing to place the data into memory 6+some 20 times. For a 1.6GB ESPRESSO dataset; this is 40 GB.
Unsurprisingly, when running this on my 16GB RAM macbook, by the time xcor() is being executed, it is using 20 GB in swap memory. Laptop then almost freezes (though seems to grind on). But as I rarely keep more than 20 GB in free disk space, this is a disaster zone.
How to address this? 1 Implement a check that deletes list_of_orders from memory after normalization, if inject_model is not called; saving 2 instances of the data in memory before. 2 Write list_of_orders and list_of_errors to drive in xcor(), then delete them from memory after hstack (sigh...), OR provide h_stacked orders to xcor to begin with (sigh...). I don't like either of these, but these would save another 2 instances of the data in memory.
These two, sadly, are peanuts compared to the casting of the template... I could implement point #1 as it is trivial to implement; but then also add a conserve-memory functionality in start_run and xcor(); where the multiplication of the template is perfomed the slow, for-loopy way; or perhaps in chunks of RV. Highly annoying.
For now, the cheapest way to conserve memory is to lower the number of RV-steps; but that limits the CCF filter-width...
Of course, this problem only occurs for template-based correlation. Template masks are tiny in memory.