search

sdss / astra

Analysis framework for SDSS-V/Milky Way Mapper
BSD 3-Clause "New" or "Revised" License
3 stars 0 forks source link

SDSS5 path differences between remote and local #3

Closed andycasey closed 4 years ago

andycasey commented 4 years ago

Astra uses sdss_access to locate paths of SDSS data products. All of the analysis will be run at Utah, but there is a separate mini-cluster that is being used for development. The mini-cluster behaviour should mimic that of Utah, in terms of path definitions, etc.

Using the sdss5 branch of the sdss/tree product, if I try to get the path of a SDSS5 apVisit file then I get something like this:

from sdss_access import SDSSPath                                                                                                                                                                                                      
path = SDSSPath(release="sdss5")                                                                                                                                                                                                      
print(path.full("apVisit", apred="daily", field="RM_XMM-LSS", telescope="apo25m", plate=15000, mjd=59146, fiber=6))
/home/andy/data/sdss/sdsswork/mwm/apogee/spectro/redux/daily/visit/apo25m/RM_XMM-LSS/15000/59146/apVisit-daily-apo25m-15000-59146-006.fits

Note the /sdsswork/mwm directory -- that's the important bit here.

The equivalent file on the SAS would be:

https://data.sdss.org/sas/sdsswork/mwm/apogee/spectro/redux/daily/visit/apo25m/RM_XMM-LSS/15000/59146/apVisit-daily-apo25m-15000-59146-006.fits

But this file does not exist! Because on the SAS the directory structure is sdss5/mwm, not sdsswork/mwm.

I don't know why the sdss5 apVisit path definition would lead me to sdsswork instead of sdss5, or why the SAS location is sdss5 instead of sdsswork, but these should be consistent.

As a temporary work-around I have edited the MWM.name definition in my local file of sdss5.cfg in the tree product to be "sdss5" instead of "sdsswork" so that I can access remote files from the SAS. But this is a hack!

@joelbrownstein and @havok2063 : this is low priority but you may want to comment.

joelbrownstein commented 4 years ago

Will fix (easy addition to the nginx conf file)!

havok2063 commented 4 years ago

@andycasey I don't actually think is a bug. sdss_access expects work paths in sdsswork. The name entry in each section of the sdss5.cfg indicates which work folder it expects to find files. And sdss_access is operating under the assumption that sdsswork is the work directory for SDSS-V, just as it was for SDSS-IV. It was expected that at the start of S-V we would switch everything over from "sdss5" to "sdsswork" but we haven't done that yet, and I think SAS directories are still pointing to a directory called "sdss5". We need a transition plan to migrate everything "sdss5"-named over to "sdsswork" and to convert the old tree sdsswork.cfg to sdss4.cfg.

joelbrownstein commented 4 years ago

@havok2063 - no it is not a bug, just an inconsistency in the SAS, now fixed! The larger issue of unifying sdsswork.cfg with sdss5.cfg can be done later (but should be done at some point).

@andycasey - please try the unhacked version now, I've setup the nginx configuration so that you can navigate, for e.g., to https://data.sdss.org/sas/sdsswork/mwm/apogee/spectro/redux (using the sdss5 username / password).

andycasey commented 4 years ago

Perfect; it works! Thank you @joelbrownstein