Open kdrushka opened 2 years ago
according to the dask client, "concat" is taking a huge amount of memory. the only concat command is combining ds and dsd (ie, adding the derived data). try just loading ds (skip dsd for now) and see if that speeds things up.
=> still slow
try xoak, which uses dask already? => xoak doesn't seem to work well. Depenging on which interpolator is used, (a) it crashes (even for only 2 days of data), or (b) it gives a nearest-neighbor answer that isn't accurate.
Try just integrating to a less fine grid than 1m - eg 5m or the model vertical grid
dask tests: 31 days {'zrange': [-1, -1000.0], 'hspeed': 0.25, 'vspeed': 0.1, 'AT_END': 'reverse', 'PATTERN': 'lawnmower', 'SAMPLING_STRATEGY': 'sim_glider', 'distance_total_km': 664.3325, 'time_total_s': 2657330.0}
tchunk = 12 xchunk = 200 ychunk = 150
186 chunks x 221.48 MiB CPU times: user 46.4 s, sys: 6.22 s, total: 52.7 s Wall time: 7min 16s
tchunk = 24 xchunk = 50 ychunk = 50
868 x 38.45 MiB CPU times: user 38.5 s, sys: 4.39 s, total: 42.9 s Wall time: 8min 58s
tchunk = 6 xchunk = 200 ychunk = 200 248 x 147.66 MiB Wall time: 3min 58s
tchunk = 3 xchunk = 250 ychunk = 250 496 = 92MB Wall time: 4min 48s
tchunk = 6 xchunk = 250 ychunk = 250 248 x 184.57MB Wall time: 3min 57s
90 days crashed
tchunk = 12 xchunk = 250 ychunk = 250 124 x 369MB Wall time: 3min 54s
tchunk = 24 xchunk = 250 ychunk = 250 162 x 738MB Wall time: 4min => crashes for 90 days
Try not loading the derived variables - where is where the concat comes from (?)