Open ali-ramadhan opened 4 years ago
It's great to have an example of such a large NetCDF. At this moment I cannot tell if this time is spent in the NetCDF C library or in the Julia wrapper code. Though I think running the slower calls under a profiler should be able to give that information.
I agree with @visr it is hard to say where the time is spent. Please note also that the NetCDF C library does some internal caching, so I guess your 3rd call was profiting from the previous reads. I found it very difficult to debug these kinds of problems. Ideally you would restart your Julia session after every data access to make sure NetCDF did not cache anything, but then you include precompilation in your timings...
i cannot reproduce with my dataset which is of similar size but only three dimensions. @ali-ramadhan is this still a problem for you?
Super cool work on integrating DiskArrays.jl with NetCDF.jl! Looking forward to ditching xarray in favor of a pure Julia solution.
@visr helped me get up and running but we noticed that grabbing 2x as much data seems to take ~4x longer whereas I expected it to scale linearly. I am unfortunately interested in grabbing data along the dimension with chunk size 1...