Slow binary installs

[timj]

Trying a beta of the new binary tarball install system created by @jhoblitt, it ended up taking 75 minutes to install lsst_distrib on my Mac. The downloading and untarring were really fast. It seems that the slow down comes from the eups table expansion in https://github.com/RobertLuptonTheGood/eups/blob/master/python/eups/distrib/Repositories.py#L767

source /Users/timj/work/tmp/binaries/eups/bin/setups.sh
export EUPS_PATH="/Users/test/work/tmp/binaries"
setup --just --type=build apr 1.5.2
setup --just --type=build cfitsio 3360.lsst4
setup --just --type=build doxygen 1.8.5.lsst1
setup --just --type=build eigen 3.2.5.lsst2
setup --just --type=build fftw 3.3.4.lsst2
setup --just --type=build gsl 1.16.lsst3
setup --just --type=build mariadbclient 10.1.21.lsst1
setup --just --type=build minuit2 5.34.14
setup --just --type=build mpich 3.2.lsst1
setup --just --type=build pybind11 master-g715ff27535
setup --just --type=build python 0.0.6
setup --just --type=build starlink_ast lsst-dev-gc4be228eb6
setup --just --type=build swig 3.0.10
setup --just --type=build xpa 2.1.15.lsst3
setup --just --type=build apr_util 1.5.4
setup --just --type=build astropy 0.0.1.lsst2+2
setup --just --type=build boost 1.60.lsst1+1
setup --just --type=build mpi 0.0.1+2
setup --just --type=build numpy 0.0.3+1
setup --just --type=build pykg_config 1.3.0+1
setup --just --type=build python_d2to1 0.2.12.lsst2+1
setup --just --type=build python_future 0.16.0
setup --just --type=build python_mysqlclient 1.3.7.lsst1+8
setup --just --type=build python_psutil 4.1.0+3
setup --just --type=build pyyaml 3.11.lsst1+3
setup --just --type=build scons 2.5.0.lsst2+1
setup --just --type=build sqlalchemy 1.0.8.lsst3+3
setup --just --type=build wcslib 5.13.lsst1
setup --just --type=build astrometry_net 0.67.123ff3e.lsst1+1
setup --just --type=build esutil 0.6.0+1
setup --just --type=build log4cxx 0.10.0.lsst7
setup --just --type=build matplotlib 0.0.4
setup --just --type=build mpi4py 2.0.0+2
setup --just --type=build ndarray 1.3.0-1-g4a984de+1
setup --just --type=build scipy 0.0.1.lsst1+3
setup --just --type=build sconsUtils 13.0-3-ga4b1818
setup --just --type=build stsci_distutils 0.3.7.lsst1+2
setup --just --type=build tmv 0.73+5
setup --just --type=build astrometry_net_data 10.0+92
setup --just --type=build astshim master-g4b275752c8+1
setup --just --type=build base 13.0-4-g0bde1de
setup --just --type=build jointcal_cholmod master-g48cb81d145
setup --just --type=build lmfit 0.9.3+4
setup --just --type=build lsst 13.0-5-g556a2c2
setup --just --type=build psfex 12.0+14
setup --just --type=build pyfits 3.4.0+7
setup --just --type=build galsim 1.4.3
setup --just --type=build healpy 1.8.1.lsst2+5
setup --just --type=build pex_exceptions 13.0-1-g52a7baa+3
setup --just --type=build utils 13.0-1-g976b40b+3
setup --just --type=build daf_base 13.0-1-g174df6e+3
setup --just --type=build geom 12.1+25
setup --just --type=build log 13.0-2-g15de9a1+3
setup --just --type=build db 13.0+7
setup --just --type=build pex_policy 13.0-1-g47a359c+3
setup --just --type=build daf_persistence 13.0-6-gf146911
setup --just --type=build pex_config 13.0-1-g41367f3+3
setup --just --type=build afw 13.0-26-g5410af9
setup --just --type=build cat 13.0-1-g532c134+7
setup --just --type=build ctrl_orca 13.0-3-g350a12c+4
setup --just --type=build ctrl_execute 13.0-2-gb581ee8+7
setup --just --type=build display_ds9 13.0-2-g6174d4c
setup --just --type=build shapelet 13.0-3-g20b541e
setup --just --type=build skymap 13.0-2-gf9e84ea
setup --just --type=build skypix 13.0+24
setup --just --type=build ctrl_platform_lsstvc master-gf8e68f9656+12
setup --just --type=build obs_base 13.0-12-g5eb5e2a
setup --just --type=build obs_monocam 13.0-2-ga69512c+19
setup --just --type=build obs_test 13.0-6-g7b63e3f
setup --just --type=build pipe_base 13.0-1-g92030ba+12
setup --just --type=build coadd_utils 13.0-2-gf5c5ced
setup --just --type=build ctrl_pool 13.0-2-gb1fa231+1
setup --just --type=build coadd_chisquared 13.0-2-ga4f5e85
setup --just --type=build meas_base 13.0-9-gd0b67f9
setup --just --type=build meas_algorithms 13.0-8-g1664ca9
setup --just --type=build meas_extensions_simpleShape 13.0-5-gaa0a428
setup --just --type=build ip_isr 13.0-9-g733b75c
setup --just --type=build meas_astrom 13.0-4-g2013b0d
eups expandtable -i --force /var/folders/ll/34nyr16s1w5_3m896tr29cvr0000gn/T/eups/distrib/meas_deblender_e7kog6uy

This runs after each package is installed. With 97 packages and it taking up to 2 minutes per table expansion this really adds up. The above meas_deblender snippet takes 75 seconds on my Mac. The initial setup calls are about 0.5 second but it ramps up to about 1.5 seconds by the end and there are 78 setup calls.

It seems silly that installing binaries should take almost as long as building from source.

@RobertLuptonTheGood, @mjuric do you have any suggestions as to how this could be sped up? Is there any way all the table expansions could run once per eups distrib install call with everything setup?

[timj]

@jdswinbank reports that the same commands takes about 38 seconds on lsst-dev01. The binary install took about 40 minutes rather than 75 minutes.

[frossie]

Ideally I'd like to see this around 15 minutes (the conda install process end to end is 12 minutes for comparison). Over an hour and as my beta tester said, "what's the point, I can use lsstsw".

[timj]

@RobertLuptonTheGood commented last week that he thinks that the expanded table files should be stored on the server rather than being generated by the client.

[RobertLuptonTheGood]

I'm planning on working on this today at NRT

    R

On 18 May 2017, at 03:38, Tim Jenness notifications@github.com wrote:

@RobertLuptonTheGood commented last week that he thinks that the expanded table files should be stored on the server rather than being generated by the client.

― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[timj]

Fix is being tested in #118.

[timj]

With #118 installed I was able to install w_2017_18 of lsst_distrib from scratch in less than 8 minutes on my laptop.