search

akutkin / frb

0 stars 0 forks source link

Refactor M5 to follow API of subsequent processing #10

Closed ipashchenko closed 8 years ago

ipashchenko commented 8 years ago

Could you refactor raw_data.py to make M5 instances to return not only numpy array with chunk of dynamical spectra, but also metadata dictionary with the following keys:n_nu, n_t, nu_0, d_nu, d_t, t_0, exp_code, antenna, freq, band ,pol? I finally merged metadata with dynamical spectra array in one class. After creating DynSpectra instances with array & metadata we don't need anything else for subsequent processing.

It could be cool if method that does this will return generator - not to store all arrays in memory but creating them from Mark5 only when they are needed in small chunks (like 5 minutes or even less).

Another option - to put data in HDF5 format that can save metadata too. See utils.save_hdf5 as example.

ipashchenko commented 8 years ago

Finally i polished dyn_spectra.DynSpectra API to level when it is the only thing we need to go further (manage pipeline etc.). Now, to proceed, i need M5 (or whatever) instances to create instances of DynSpectra class (one instance for each chunk of Mark5 file). This part is for you, @akutkin ;) I think it would be easy cause DynSpectra creating needs only dyn. spectra array + metadata with fixed keys.

ipashchenko commented 8 years ago

resovled with 2866a0f3029c63a49703f3a8a0890b41c268fd73