search

RadioAstronomySoftwareGroup / pyuvdata

A pythonic interface for radio astronomy interferometry data (uvfits, miriad, others)
https://pyuvdata.readthedocs.io/en/latest/index.html
BSD 2-Clause "Simplified" License
82 stars 27 forks source link

mwa_corr_fits does not support metadata only reads #830

Closed mwilensky768 closed 4 years ago

mwilensky768 commented 4 years ago

Using /usr/bin/time -v I've found that attempting a metadata-only read for 10G worth of MWA corr fits files takes about 50G of RAM. This seems unnecessary. It also appears that UVData.read_mwa_corr_fits does not have a read_data keyword, which may be the culprit i.e. the metadata-only functionality is not there to begin with.

mkolopanis commented 4 years ago

is the 50GB usage consistent with reading the whole file?

bhazelton commented 4 years ago

Oh, yeah, reading only the metadata isn't an option (yet) for mwa_corr_fits files, so that keyword is just being ignored.

mkolopanis commented 4 years ago

There are some other keywords like the check related that do not get used. Maybe I should add this stuff in as part of the #825 ?

mwilensky768 commented 4 years ago

@mkolopanis the data array is indeed actually read in (because read_data=False is ignored). I would assume that an ignored keyword isn't changing memory consumption and that this is consistent with reading the whole file in with read_data=True.

mwilensky768 commented 4 years ago

Just for the record, it seems reading in a similar-sized uvfits file metadata-only style does not have large memory demands, so I think this is strictly caused by not actually doing metadata-only for mwa_corr_fits. I will change the title of the issue.