Reorganize output into logical subfolders

[bhazelton]

This branch reorganizes the output of eppsilon into the following folders:

• ps

  • data

  • uvf_cubes

    • slices

  • kspace_cubes

    • slices
    • beams
    • 2d_binning
    • from_1d
    • 1d_binning (includes kperp/kpar files & kpar=0 files)

  • plots

  • slices

  • 2d_binning

    • from_1d

  • 1d_binning

    • bin_histograms

It is not yet backwards compatible, I have some questions about what users want for backwards compatibility. I think there are two options:

1. When eppsilon is called on an old folder it can reorganize that folder according to this structure. If we do this, the easiest implementation will only move the files associated with the run it's called for, so if there are files generated by runs using different keyword settings, those files would not be moved until eppsilon is re-run with those settings. If this is terrible, I can try to write something that generally moves all the files, but it'll be harder. Also, I wonder whether the existing plot files should just all be deleted (since they are re-created on each run) or whether I should test for and delete only the plotfiles associated with the run it's called for.
2. When eppsilon is called on an old folder it just leaves the existing files in place and only writes new files (and plots) to the new structure.

I'd like to get feedback from users about which of these options they prefer. @adampbeardsley @nicholebarry @rlbyrne @mkolopanis

[bhazelton]

fixes issue #19

[rlbyrne]

My vote is for 2. 1 sounds like it could get confusing fast.

[mkolopanis]

The option 1 sounds like it would be confusing work from a development side. If a users folder is so large that option 2 causes massive bloating I think it is reasonable to assume they can handle removing files as necessary.

[adampbeardsley]

I'm in agreement with Matt and Ruby. Having a program dynamically find files and move them around makes me nervous.

[wenyang-li]

I agree with everyone upstairs. I prefer option 2 too.

[bhazelton]

Ok, thank you all for the input. I believe I have it all working properly now, including for difference and ratio plots.

When you re-run ps_wrapper on an old folder, the DFT and power spectra should not be recalculated, but it will redo the binning (which is pretty fast). Existing files will not be moved, but any new files that are written will be written to the new locations. Files that are read but not recalculated will be left in the old location and not copied to the new location.

Difference and Ratio plots should run on any combination of new and old folders with no need to rerun anything.

I have tested it some on my own, but I'd love for some volunteers to do some real-world testing and make sure I haven't added any new bugs.

[bhazelton]

Any objections to my merging this? @nicholebarry @rlbyrne @mkolopanis @wenyang-li @adampbeardsley

[adampbeardsley]

I say go for it! On Tue, Feb 27, 2018 at 4:02 PM Bryna Hazelton notifications@github.com wrote:

Any objections to my merging this? @nicholebarry https://github.com/nicholebarry @rlbyrne https://github.com/rlbyrne @mkolopanis https://github.com/mkolopanis @wenyang-li https://github.com/wenyang-li @adampbeardsley https://github.com/adampbeardsley

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/EoRImaging/eppsilon/pull/44#issuecomment-369058861, or mute the thread https://github.com/notifications/unsubscribe-auth/AEiGFU9fQmZXVrF1VdvvoUT0rQjFjgf2ks5tZImWgaJpZM4RsMdc .

-- Adam Beardsley National Science Foundation Fellow Arizona State University School of Earth and Space Exploration Low Frequency Cosmology Homepage: loco.lab.asu.edu/adam_beardsley

[rlbyrne]

Fine by me!

[wenyang-li]

Ditto.