Closed sbailey closed 1 year ago
weaverba137
commented
1 year ago From my experiments, we can at least hide the values of the "Don't Add"s by using pushenv and pushing an empty value:
pushenv TYCHO_DIR ""and then they should be restored as expected.
We could also leave these alone, but it is sort of advertising the existence of things we don't necessarily want people asking us about.
weaverba137
commented
1 year ago For the record: DESI_BASIS_TEMPLATES and DESI_MASK are now in place.
weaverba137
commented
1 year ago Note, the implied path of QN_MODEL_FILE actually puts it in ets not edr because edr/target/catalogs is a pure symlink to ets/target/catalogs. This is OK for this one file, but, e.g. dr1/target/catalogs may need to be an actual directory with subdirectories linking to ets.
sbailey
commented
1 year ago @weaverba137 I added the surveyops sv2 and sv3 directories with the following commands as the desi user; these are ready for you to add checksums (after re-granting write access) and then you can close this.
# cd and re-grant ability to write to this dir
cd $CFS/desi/public/edr/survey/ops/surveyops/tags/0.1/mtl
chmod u+w .
# svn export sv2 and sv3
svn export https://desi.lbl.gov/svn/data/surveyops/tags/0.1/mtl/sv2
svn export https://desi.lbl.gov/svn/data/surveyops/tags/0.1/mtl/sv3
# grant world read + execute
chmod -R a+rX sv2
chmod -R a+rX sv3
# remove write permission, even from self
chmod -R a-w .For the record: we will leave the environment variables pointing to the proprietary locations, which might have some use for DESI collaborators switching to desitree/edr . We can revisit that choice if it turns out to be problematic.
For documentation purposes, "module swap desitree/edr" appears to be the standard module syntax, though "module switch desitree/edr" appears to do the same thing. e.g. see https://lmod.readthedocs.io/en/latest/010_user.html which documents "swap" but not "switch".
@weaverba137 recreated the checksum file; closing the ticket.
weaverba137
commented
1 year ago Also note: the resulting edr module file is archived as etc/edr.module.
This is related to the environment variables in desihub/desimodules#47, but I'm starting a new ticket to focus on EDR data instead of software environments, and I'm putting it in desitree as the closest repo we have for data directory organization topics.
ADD TO EDR
Add the following directories to EDR and update desitree/edr to set these to the EDR location:
ADD TO EDR
These further augment $DESI_SURVEYOPS=/global/cfs/cdirs/desi/public/edr/survey/ops/surveyops/tags/0.1/ These exist in the tag, but not yet in the export (we only want to release a subset of the tag).
DON'T ADD
For the record, some other directories that we should not add to EDR. If we can find an easy way for desitree/edr to unset these but have them re-appear when swapping back to desitree/0.6.0 that would be nice. If that is tricky, we could leave them as-is (which means they would continue to work for DESI collaborators which is sort of a feature rather than a bug).
Reformatting of other surveys' data releases
These are needed as inputs for running target selection but not needed for using target selection outputs. Re-releasing them may have licensing / acknowledgements that's we'd have to clarify first
Working directories for survey ops
(and in some cases don't even exist because env is redefined elsewhere for real usage)
Will add to DR1
We'll want it for DR1, but it needs some cleanup first. Caveat: we may want to post-facto add it to EDR just to support more recent software releases, but even then it is only used for low level CCD preprocessing.
Unused