kkappler
commented
2 months ago @kujaku11 I made a minor change to transfer_function_kernel.py so that the function that looks for the Fourier coefficients does so in read mode. Can you check of this makes a significant performance improvement?
Also, could you tell if the slowdown was linear, or did it get worse over time? i.e. did looping the second half of the runs take about the same time as the first half? or was it noticeably slower near the end of the process?
TLDR:
The intention of the decorater was two things:
- To allow cleaner code for methods that interact with mth5, removing the need to apply context manager, importing mth5, open and close in every function.
- To allow some simple interactive calls in other programs that don't risk leaving an open mth5 around.
I ran into a similar issue with a dataset having hundreds of runs. Decorator abuse did seem to be the issue -- specifically nesting calls to functions with this the decorator was the likely culprit. i.e. if it is used once with a filename, then subsequent uses should pass the mth5_object, not the filepath.
That issue however appeared in the form of multi-hour slowdowns, but I believe that your issue was on the order of minutes. I wonder of this is being caused by recomputing the channel summary at each call to close_mth5()? If so, opening the file in read mode should help a lot.
The decorator takes a kwarg for mode, which if set to "r" should close the file without recomputing the channel summary.
While processing short burst of data, could be up to 400 1 or 2s bursts, there is a slow down in Aurora. Likely caused by the open_mth5 decorator. Because each burst opens the MTH5 gets the run then closes.