zpace
commented
5 years ago I have run a quick test with DRP LOGCUBE instead of the DAP LOGCUBE (just b/c that's what it's handiest for me to script), and I'll try to summarize. I wrote a couple quick functions that respectively load through a bare wrapper around fits.open() and marvin.tools.Cube(), and profiled the result.
def get_fluxcube_fits(plate, ifu, MPL_V):
'''a wrapper around fits.open()
'''
with m.load_drp_logcube(plate, ifu, MPL_V) as drp: # can replace with fits.open(drp_logcube_filename)
return drp['FLUX'].data
def get_fluxcube_marvin(plate, ifu, **kwargs):
'''a wrapper around marvin.tools.Cube
'''
drp_cube = marvin.tools.Cube('-'.join((plate, ifu)), **kwargs)
return drp_cube.flux
def compare_two_loads(plate, ifu, MPL_V):
'''runs both loaders
'''
fits_cube = get_fluxcube_fits(plate, ifu, MPL_V)
marvin_cube = get_fluxcube_marvin(plate, ifu, release=MPL_V)
return fits_cube, marvin_cube
%prun -D loadcube.prof compare_two_loads(plate, ifu, MPL_V)I then loaded loadcube.prof (which github won't let me upload here) into snakeviz.
Here's a screengrab:
What jumped out at me is the large number of calls to zlib.Decompress.decompress (about 2/3 of the time taken by get_fluxcube_marvin()). Jose and I had talked in Ensenada about fits files having to be decompressed prior to use, so this makes sense. IIRC, individual HDUs are decompressed as needed.
Just to verify this, I wrote another bare wrapper that accessed every HDU of the LOGCUBE in a list comprehension, and profiled it:
def get_alldata_fits(plate, ifu, MPL_V):
with m.load_drp_logcube(plate, ifu, MPL_V) as drp:
data = [hdu.data for hdu in drp]
return data
def compare_two_fullloads(plate, ifu, MPL_V):
fits_data = get_alldata_fits(plate, ifu, MPL_V)
marvin_cube = get_fluxcube_marvin(plate, ifu, release=MPL_V)And sure enough, get_all_data_fits() takes 30-ish seconds. The culprit is once again the zlib decompression. I think users with lots of hard-disk space could profit from pre-extracting all the data files they plan to use, if marvin could "prioritize" ingesting uncompressed files over compressed files.
If there's any other tests that would be informative, please do let me know!
havok2063
@zpace reported that doing batch operations with
ModelCubeloaded from files seems to be slower as compared with using rawastropy.fits. He agree to distill the issue to a simple script that we can test. We should profile the file loading and see if there is any obvious bottleneck. This is somehow related to #298.