Closed aelanman closed 5 years ago
I can confirm that this would be useful. New cluster that I'm on has a file limit...
I'm trying to figure out what the interface for this should look like. It's easy to stop after making the 3D power cubes. What I'm having a hard time figuring out is what the interface should be for requesting specific 1 or 2d files.
I don't know how feasible this would be, but maybe have it be that if the "lightweight_mode" keyword is set, it will only do 1d or 2d binning if explicitly requested.
It'd be like a variation on the "plot_stdset" option?
Ok, that's easy to do. But when you ask it to do the binning it will still make all the normal files. Is that what you want?
That should be fine. If I can be of any help, please let me know.
I'm thinking of raising another issue too, but it might be related... it'd be nice if there were a way to rerun eppsilon without reference to the Healpix cubes (ie., just rebinning existing power cubes), but Nichole says there doesn't seem to be a way yet.
Ok, I'll implement that (I've already started). Do you also want the option to turn off saving slices of the 3d cubes? That seems like low-hanging fruit for limiting the number of files.
On the other question, I think that once you've made an info file, uvf cubes and the beam files you shouldn't need to have access to the Healpix files. I know Nichole had a problem making that work in the past, but I thought I'd fixed that bug. If it doesn't work, please make an issue detailing the problem and I'll try to fix it.
Thanks! Yes, if we could turn off the slices too that could cut down on file generation. I had trouble turning off the horizon cuts. I ended up just writing a command into my wrapper that removed them after the fact.
Once I set uvf_input, it ran just fine without the Healpix cubes.
I mentioned this in the telecon, but the uvf_input
keyword does NOT indicate using the eppsilon calculated uvf cubes, rather it indicates that you want to use the FHD gridded uvf cubes (i.e. before going to healpix). I welcome updates to the docs to make that clearer.
Thanks for letting me know. The documentation did clarify that well. I think I just misunderstood Nichole. I will make sure things still run without the Healpix cubes present.
@aelanman @nicholebarry please try out the branch in PR #80 and let me know if it meets your needs.
There should be a way of running eppsilon that doesn't calculate diagnostic figures or save a lot of metadata. Specifically, it'd be nice if it could only calculate uvf, power, and k cubes, and then produce specific 1D or 2D binned spectra only if requested.
I'm trying to run a large number of Obs IDs through eppsilon in batch, and without integrating the observations, and I'm concerned about file bloat.