search

openghg / openghg_inversions

University of Bristol Atmospheric Chemistry Research Group RHIME Inversion code (with openghg dependency)
MIT License
5 stars 0 forks source link

`local_influence` filter potentially very slow #180

Open brendan-m-murphy opened 4 months ago

brendan-m-murphy commented 4 months ago

In Issue #179, @qq23840 noted that the local_influence filter was running for 30 minutes without finishing.

The bug in Issue #176 is addressed by PR #179, but the performance issue isn't addressed by that PR.

qq23840 commented 4 months ago

The issue came in the reading of dask-arrays in a loop in _local_ratio. It was taking 5-6 seconds to read from disk for every measurement, meaning a whole year's worth of data (say 1500 measurements) would take hours to filter. Fixed by adding a dataset.load() line before the loop starts. Won't add this to devel as I think it messes with some CO2 stuff, but putting here for reference