Lightweight mode

[aelanman]

There should be a way of running eppsilon that doesn't calculate diagnostic figures or save a lot of metadata. Specifically, it'd be nice if it could only calculate uvf, power, and k cubes, and then produce specific 1D or 2D binned spectra only if requested.

I'm trying to run a large number of Obs IDs through eppsilon in batch, and without integrating the observations, and I'm concerned about file bloat.

[nicholebarry]

I can confirm that this would be useful. New cluster that I'm on has a file limit...

[bhazelton]

I'm trying to figure out what the interface for this should look like. It's easy to stop after making the 3D power cubes. What I'm having a hard time figuring out is what the interface should be for requesting specific 1 or 2d files.

[aelanman]

I don't know how feasible this would be, but maybe have it be that if the "lightweight_mode" keyword is set, it will only do 1d or 2d binning if explicitly requested.

It'd be like a variation on the "plot_stdset" option?

[bhazelton]

Ok, that's easy to do. But when you ask it to do the binning it will still make all the normal files. Is that what you want?

[aelanman]

That should be fine. If I can be of any help, please let me know.

I'm thinking of raising another issue too, but it might be related... it'd be nice if there were a way to rerun eppsilon without reference to the Healpix cubes (ie., just rebinning existing power cubes), but Nichole says there doesn't seem to be a way yet.

[bhazelton]

Ok, I'll implement that (I've already started). Do you also want the option to turn off saving slices of the 3d cubes? That seems like low-hanging fruit for limiting the number of files.

On the other question, I think that once you've made an info file, uvf cubes and the beam files you shouldn't need to have access to the Healpix files. I know Nichole had a problem making that work in the past, but I thought I'd fixed that bug. If it doesn't work, please make an issue detailing the problem and I'll try to fix it.

[aelanman]

Thanks! Yes, if we could turn off the slices too that could cut down on file generation. I had trouble turning off the horizon cuts. I ended up just writing a command into my wrapper that removed them after the fact.

Once I set uvf_input, it ran just fine without the Healpix cubes.

[bhazelton]

I mentioned this in the telecon, but the uvf_input keyword does NOT indicate using the eppsilon calculated uvf cubes, rather it indicates that you want to use the FHD gridded uvf cubes (i.e. before going to healpix). I welcome updates to the docs to make that clearer.

[aelanman]

Thanks for letting me know. The documentation did clarify that well. I think I just misunderstood Nichole. I will make sure things still run without the Healpix cubes present.

[bhazelton]

@aelanman @nicholebarry please try out the branch in PR #80 and let me know if it meets your needs.