To-do list for September 7 Busy Day - restructuing data storage paradigm

[bwmeyers]

The primary goal of the day is to adjust our current software workflows to move away from using the communal /astro/mwavcs/vcs directory to house data, calibration solutions, beamformer products, etc.The new scheme will be: data and products are stored in the user's /astro/mwavcs/$USER space. This will help to mitigate our "phantom data" issues and, hopefully, get people in the habit of deleting data when they no longer need it on disk. It will notionally make data tracking easier, too.

Here, I've listed some high-level tasks. We will likely have to split some of these up into sub-tasks dealing with specific pieces of code:

• [x] Clean up ASVO space of old data.
• [ ] Have users clean up their data from the communal space.
• [ ] Save any necessary data/products to Acacia.
• [ ] Delete observation directories within /astro/mwavcs/vcs. (This will probably take more than just one day to complete.)
• [x] Remove hard-coded communal path from vcstools scripts/configuration files and replace with user-specific directories.
• [ ] #240

[bwmeyers]

We can, and should, split things out into separate issues as appropriate, but this may be better suited as a first agenda item of the Busy Day,

[robotopia]

Dodged a bullet putting this on 7 Sep instead of 6 Sep, 'coz of monthly maintenance. Fingers crossed it all goes smoothly!

[robotopia]

grep -r "/astro/mwavcs/vcs" *

scripts/combine_calibrations.py:        comp_config = {'base_data_dir' : "/astro/mwavcs/vcs/"}
scripts/rename_corr_output.py:        workdir='/astro/mwavcs/vcs/{0}/vis'.format(obsID)
scripts/rename_corr_output.py:                            "that contains the output files of the offline Correlator. Default is /astro/mwavcs/vcs/<obsID/vis.")
scripts/checks.py:        comp_config = {'base_data_dir' : "/astro/mwavcs/vcs/"}
scripts/untar.sh:\t\t (default: /astro/mwavcs/vcs/\${obsID}/combined)\n
scripts/untar.sh:\t\t (default: /astro/mwavcs/vcs/\${obsID}/combined)\n
scripts/untar.sh:    workdir=/astro/mwavcs/vcs/${obsid}/combined/
scripts/untar.sh:    outdir=/astro/mwavcs/vcs/${obsid}/combined
scripts/recombine.py:    parser.add_argument("-p", "--output_base_dir", type=str, help="The base directory to put the recombined data. Eg /astro/mwavcs/vcs/<obsid>.")
scripts/process_vcs.py:    # Downloads the visablities to  /astro/mwavcs/vcs/[cal_obs_id]/vis
scripts/process_vcs.py:    # but creates a link to it here /astro/mwavcs/vcs/[obs_id]/cal/[cal_obs_id]
scripts/process_vcs.py:                          "raw data will will be downloaded into /astro/mwavcs/vcs/[obsID] and data products will be in /group/mwavcs/vcs/[obsID]."
scripts/reorder_chans.py:    parser.add_argument("-m", "--mfits", type=str, help="Please provide the path of the *_metafits_ppds.fits if it is not /astro/mwavcs/vcs/[OBID]/*_metafits_ppds.fits")
scripts/reorder_chans.py:        metafits = '/astro/mwavcs/vcs/' + str(args.obsid) + '/' + str(args.obsid) + '_metafits_ppds.fits'
vcstools/check_files.py:        |br| Default: /astro/mwavcs/vcs/[obsID]/raw/.
vcstools/check_files.py:        |br| Default: /astro/mwavcs/vcs/[obsID]/combined/.
vcstools/check_files.py:        |br| Default: /astro/mwavcs/vcs/[obsID]/combined/.
vcstools/config.py:GALAXY_CONFIG = {'base_data_dir' : '/astro/mwavcs/vcs/',
vcstools/config.py:GARRAWARLA_CONFIG = {'base_data_dir' : '/astro/mwavcs/vcs/',