Please install the latest desiconda on datatran

[weaverba137]

The datatran software stack shares desimodules with cori and edison, so now it is looking for a version of desiconda that is not installed. As a result, certain automated processes are not running with the correct version of Python.

desimodules/17.6(51):ERROR:105: Unable to locate a modulefile for 'desiconda/20170613-1.1.4-spectro'
desimodules/17.6(51):ERROR:151: Module 'desimodules/17.6' depends on one of the module(s) 'PW'
desimodules/17.6(51):ERROR:102: Tcl command execution failed: if { [info exists env(NERSC_HOST)] } {
    module load desiconda/20170613-1.1.4-spectro
    prereq desiconda
}

Traceback (most recent call last):
  File "/global/homes/b/bweaver/software/code/datatran/decamUtil/trunk/bin/transfer_noao_to_nersc", line 3, in <module>
    from decamUtil.transfer import main
  File "/global/homes/b/bweaver/software/code/datatran/decamUtil/trunk/py/decamUtil/transfer.py", line 18, in <module>
    from shutil import chown
ImportError: cannot import name chown

[tskisner]

datatran will need a special solution, since it does not have a modern compiler. I don't think we need any of the compiled packages (cfitsio, boost, fftw, healpix) for data transfer operations, correct?

[weaverba137]

I'm pretty sure fitsio/cfitsio is necessary.

[tskisner]

fitsio (and healpy) can be pip-installed. I'll add a "minimal" config to desiconda which does not build any compiled things.

[tskisner]

Hello @weaverba137 , could you look at: /project/projectdirs/desi/software/datatran/desiconda/20170818-1.1.12 and see if that meets your needs?

[weaverba137]

The startup module files are not setup on datatran, so this isn't 100% ready to go. I can set that up and then test again.

[weaverba137]

I can't find cfitsio in the $DESICONDA_EXTRA directory. Please install the cfitsio C library on datatran. This is really important, as it's needed to build fitsverify.

[weaverba137]

Where is the code directory? It should be in the same directory as the modulefiles directory. This version of desiconda doesn't have desiutil installed, & that's also really important.

Can you please make the desiconda install as exactly the same as cori and edison as possible?

[tskisner]

@weaverba137 , I thought this ticket was about desiconda? desiUtil and all the other desi packages are completely separate from desiconda. desiconda consists only of desi "dependencies". This means the python stack and a set of compiled external packages. In other words, desiconda is "scripts to install software that is external from DESI". So I believe that all these external packages are now installed on all the machines in a consistent manner.

The next step is to install all the DESI-maintained packages using these external dependencies. However, that step has nothing to do with desiconda. For example, the imaging survey people use desiconda as-is. I use the /global/common installed desiconda when I'm working with my own checkouts of DESI software. Not sure which git repo the issue should be opened on to "update the nersc installs of desi code". Maybe desimodules, since that is where the versions of packages are tracked?

[weaverba137]

OK, I get that desiconda does not include the DESI packages. However, there is still the issue of missing libraries. I can't find cfitsio or sqlalchemy. Both are essential.

[weaverba137]

For sqlalchemy see #10.

[tskisner]

Ok, I thought only the python fitsio package (not libcfitsio) was needed. fitsio is pip-installed. If we are not installing specex and fiberassign (which presumably we don't need for data transfer), which other desi packages require cfitsio?

[weaverba137]

fitsverify

[tskisner]

OK, thanks for clarifying. I'll install the cfitsio package as part of the "minimal" desiconda install. This should be fine on datatran, since cfitsio does not require C++11. The lack of C++11 is the reason we can't install boost, harp, etc (and thus the "full" spectro desiconda install) on datatran.

[weaverba137]

Good. And fitsverify itself also only requires a basic C compiler, not anything especially recent.

[tskisner]

All changes to the code requested by this ticket should be in place as of a61749e568be4439a0cb615d790f645d04f892a1. However, I will leave this ticket open until I can actually reinstall a new desiconda version everywhere.

[tskisner]

This should be fully resolved now that 1.2.2 is installed.