search

deeplycloudy / xlma-python

A future, Python-based version of xlma?
MIT License
6 stars 6 forks source link

add flash sort + grid to netcdf example #23

Closed wx4stg closed 9 months ago

wx4stg commented 9 months ago

I figured it would be a good idea to create an example script of how one could read in VHF source data, sort into flashes and grid using pyxlma, and then write out to a netcdf file for other tools to use.

...however, this currently produces netcdf files in the range of ~6GiB for 10 mins of data from TRACER. I think something I'm doing here is inefficient so I'll need to look into this more.

wx4stg commented 9 months ago

adding the compression flag is a good thing, as it turns out :)

wx4stg commented 9 months ago

profiling commit d16e2e7, I get about a 1.5ms performance hit on an Apple M2 Max.

I don't think this is too terrible for the functionality it provides and it can be disabled.

Test data used attached. image

LYLOUT_220730_223000_0600.dat.gz

wx4stg commented 9 months ago

for a rather busy one hour of data, it's a ~2ms hit if the data is already sorted, and a ~15ms hit if the data is randomly disorganized beforehand

image

LYLOUT_210409_010000_3600-sorted.dat.gz

LYLOUT_210409_010000_3600-unsorted.dat.gz

deeplycloudy commented 9 months ago

Thanks, @wx4stg, for the changes in response to in-person conversation. Merging, and looking forward to community use of this.