search

mjuric / conda-lsst

Building Conda packages for the LSST stack
4 stars 8 forks source link

$PWD/miniconda/bin also needs to be in PATH #45

Open johannct opened 8 years ago

johannct commented 8 years ago

else the correct python framework is not selected against the local one.

johannct commented 8 years ago

miniconda/bin/conda lsst make-recipes build:b1852 lsst_distrib lsst_sims --build fails when trying to build the first package lsst-doxygen. The log points to failing to run /bin/bash -x -e /home/cohen/lsst/dm/conda-lsst/recipes/lsst-doxygen/build.sh which indeed returns : ++ git rev-parse --show-toplevel

mjuric commented 8 years ago

Hi @johannct, That's unexpected, as eups-setups.sh should automatically be installed in the _build environment, picked up as a dependency (you shouldn't have to have it installed in your root environment).

Could you send me the full log from the doxygen failure, and also the output of the conda lsst make-recipes .... --build command?

johannct commented 8 years ago

I tried to relaunch it, but now I have yet another error : cohen@port-jct:~/lsst/dm/conda-lsst$ echo $PATH /home/cohen/lsst/dm/conda-lsst/miniconda/bin:/home/cohen/lsst/dm/conda-lsst/bin:/home/cohen/sources/astro/Healpix_3.30/src/cxx//bin:/home/cohen/sources/astro/Healpix_3.30/bin:/home/cohen/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/anaconda1anaconda2anaconda3/opt/eups/bin cohen@port-jct:~/lsst/dm/conda-lsst$ conda lsst make-recipes build:b1852 lsst_distrib lsst_sims --build updating built package cache [from file:///home/cohen/lsst/dm/conda-lsst/miniconda/conda-bld/linux-64] done. updating built package cache [from http://eupsforge.net/conda/dev/linux-64] ..+.+.+++++.++++.++.+.+++++++.+.+++++++++++++.++++..+++.+++.++++++++++++++++.++.++.+.++++++.++.+++.++++++.++++.++++.+++.+++++++..+.+++++++++++++++++.+++.+++++.+.++++++.+.++.++..++++++..+.+..+++.++Traceback (most recent call last): File "/home/cohen/lsst/dm/conda-lsst/bin/conda-lsst", line 147, in db.reindex(config.channels) File "/home/cohen/lsst/dm/conda-lsst/conda_lsst/recipe_db.py", line 186, in reindex self.reindex_channel(channel) File "/home/cohen/lsst/dm/conda-lsst/conda_lsst/recipe_db.py", line 263, in reindex_channel all = tf.getnames() File "/home/cohen/lsst/dm/conda-lsst/miniconda/lib/python2.7/tarfile.py", line 1844, in getnames return [tarinfo.name for tarinfo in self.getmembers()] File "/home/cohen/lsst/dm/conda-lsst/miniconda/lib/python2.7/tarfile.py", line 1836, in getmembers self._load() # all members, we first have to File "/home/cohen/lsst/dm/conda-lsst/miniconda/lib/python2.7/tarfile.py", line 2416, in _load tarinfo = self.next() File "/home/cohen/lsst/dm/conda-lsst/miniconda/lib/python2.7/tarfile.py", line 2347, in next self.fileobj.seek(self.offset - 1) EOFError: compressed file ended before the logical end-of-stream was detected

Surprisingly it tries to recache everything here, so maybe something changed on the server? Anyway I will try a complete reinstall (with the ./local easy_install.pth kludge and adding miniconda/bin to PATH) and will report back

johannct commented 8 years ago

Ok here is the full screen output of my clean install attempt : cohen@port-jct:~$ cd lsst/dm/ cohen@port-jct:~/lsst/dm$ git clone git@github.com:mjuric/conda-lsst.git Cloning into 'conda-lsst'... remote: Counting objects: 499, done. remote: Total 499 (delta 0), reused 0 (delta 0), pack-reused 498 Receiving objects: 100% (499/499), 158.89 KiB | 0 bytes/s, done. Resolving deltas: 100% (245/245), done. Checking connectivity... done. cohen@port-jct:~/lsst/dm$ cd conda-lsst/ cohen@port-jct:~/lsst/dm/conda-lsst$ export PATH=$PWD/bin:^C cohen@port-jct:~/lsst/dm/conda-lsst$ ls bin conda_lsst etc README.md samples scripts cohen@port-jct:~/lsst/dm/conda-lsst$ bin/bootstrap.sh % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 24.2M 100 24.2M 0 0 906k 0 0:00:27 0:00:27 --:--:-- 1037k PREFIX=/home/cohen/lsst/dm/conda-lsst/miniconda installing: _cache-0.0-py27_x0 ... installing: python-2.7.11-0 ... installing: conda-env-2.4.5-py27_0 ... installing: openssl-1.0.2d-0 ... installing: pycosat-0.6.1-py27_0 ... installing: pyyaml-3.11-py27_1 ... installing: readline-6.2-2 ... installing: requests-2.9.0-py27_0 ... installing: sqlite-3.8.4.1-1 ... installing: tk-8.5.18-0 ... installing: yaml-0.1.6-0 ... installing: zlib-1.2.8-0 ... installing: conda-3.19.0-py27_0 ... installing: pycrypto-2.6.1-py27_0 ... installing: pip-7.1.2-py27_0 ... installing: wheel-0.26.0-py27_1 ... installing: setuptools-18.8.1-py27_0 ... Python 2.7.11 :: Continuum Analytics, Inc. creating default environment... installation finished. Fetching package metadata: ...... Solving package specifications: ........................ Package plan for installation in environment /home/cohen/lsst/dm/conda-lsst/miniconda:

The following packages will be downloaded:

package build
openssl-1.0.2f 0 3.2 MB
patchelf-0.8 0 258 KB
sqlite-3.9.2 0 3.9 MB
markupsafe-0.23 py27_0 30 KB
requests-2.9.1 py27_0 605 KB
setuptools-19.6.2 py27_0 365 KB
sqlalchemy-1.0.11 py27_0 1.3 MB
conda-3.19.1 py27_0 177 KB
jinja2-2.8 py27_0 263 KB
pip-8.0.2 py27_0 1.5 MB
conda-build-1.19.0 py27_0 211 KB 
------------------------------------------------------------
                                       Total:        11.7 MB

The following NEW packages will be INSTALLED:

conda-build: 1.19.0-py27_0
jinja2:      2.8-py27_0   
markupsafe:  0.23-py27_0  
patchelf:    0.8-0        
sqlalchemy:  1.0.11-py27_0

The following packages will be UPDATED:

conda:       3.19.0-py27_0 --> 3.19.1-py27_0
openssl:     1.0.2d-0      --> 1.0.2f-0     
pip:         7.1.2-py27_0  --> 8.0.2-py27_0 
requests:    2.9.0-py27_0  --> 2.9.1-py27_0 
setuptools:  18.8.1-py27_0 --> 19.6.2-py27_0
sqlite:      3.8.4.1-1     --> 3.9.2-0

Fetching packages ... openssl-1.0.2f 100% |################################| Time: 0:00:05 638.08 kB/s patchelf-0.8-0 100% |################################| Time: 0:00:00 321.44 kB/s sqlite-3.9.2-0 100% |################################| Time: 0:00:05 739.08 kB/s markupsafe-0.2 100% |################################| Time: 0:00:00 172.33 kB/s requests-2.9.1 100% |################################| Time: 0:00:01 504.23 kB/s setuptools-19. 100% |################################| Time: 0:00:01 268.56 kB/s sqlalchemy-1.0 100% |################################| Time: 0:00:02 455.35 kB/s conda-3.19.1-p 100% |################################| Time: 0:00:00 210.42 kB/s jinja2-2.8-py2 100% |################################| Time: 0:00:00 392.95 kB/s pip-8.0.2-py27 100% |################################| Time: 0:00:03 505.93 kB/s conda-build-1. 100% |################################| Time: 0:00:00 246.62 kB/s Extracting packages ... [ COMPLETE ]|###################################################| 100% Unlinking packages ... [ COMPLETE ]|###################################################| 100% Linking packages ... [ COMPLETE ]|###################################################| 100% Collecting requests-file Collecting six (from requests-file) Using cached six-1.10.0-py2.py3-none-any.whl Requirement already satisfied (use --upgrade to upgrade): requests>=1.0.0 in ./miniconda/lib/python2.7/site-packages (from requests-file) Installing collected packages: six, requests-file Successfully installed requests-file-1.4 six-1.10.0 Miniconda has been installed in /home/cohen/lsst/dm/conda-lsst/miniconda. Add it to your path:

export PATH="/home/cohen/lsst/dm/conda-lsst/miniconda/bin:$PATH"

and continue. cohen@port-jct:~/lsst/dm/conda-lsst$ export PATH=/home/cohen/lsst/dm/conda-lsst/bin:/home/cohen/lsst/dm/conda-lsst/miniconda/bin:$PATH cohen@port-jct:~/lsst/dm/conda-lsst$ conda config --add channels http://eupsforge.net/conda/dev Skipping channels: http://eupsforge.net/conda/dev, item already exists cohen@port-jct:~/lsst/dm/conda-lsst$ conda lsst make-recipes build:b1852 lsst_distrib lsst_sims --build updating built package cache [from file:///home/cohen/lsst/dm/conda-lsst/miniconda/conda-bld/linux-64] not found. skipping. updating built package cache [from http://eupsforge.net/conda/dev/linux-64] ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ done. https://raw.githubusercontent.com/lsst/versiondb/master/manifests/b1852.txt generating recipes: lsst-doxygen-1.8.5.1...
eups-1.5.9...
lsst-sconsutils-2015_11.0.2.post1...
lsst-product-configs-0.0.1...
lsst-eups-environment-2015_10.0.post11...
lsst-boost-1.59.5...
lsst-base-2015_10.0.2...
lsst-pex-exceptions-2015_10.0.1.post11...
lsst-utils-2015_10.0.3.post9...
lsst-daf-base-2015_10.0.3.post9...
lsst-mysqlclient-5.1.73.3...
lsst-pex-logging-2015_10.0.1.post11...
lsst-pex-policy-2015_10.0.1.post12...
lsst-python-d2to1-0.2.12...
lsst-stsci-distutils-0.3.7.1...
lsst-pyfits-3.2.4.2.post1...
lsst-daf-persistence-2015_10.0.6.post5...
lsst-pex-config-2015_10.0.1.post11...
lsst-eigen-3.2.5...
lsst-fftw-3.3.4.2...
lsst-ndarray-0.10.1.post55...
lsst-cfitsio-3360.4...
lsst-wcslib-4.14.2...
lsst-minuit2-5.28.00.2...
lsst-gsl-1.16.3...
lsst-afw-2015_10.0.14...
lsst-geom-0.10.0.post47...
lsst-skypix-0.10.0.post330...
lsst-daf-butlerutils-2015_10.0.4.post1...
lsst-obs-test-2015_10.0.3.post6...
lsst-pipe-base-2015_10.0.5...
lsst-coadd-utils-2015_10.0.4.post11...
lsst-meas-base-2015_10.0.4.post11...
lsst-meas-algorithms-2015_10.0.4.post3...
lsst-meas-deblender-2015_10.0.3.post22...
lsst-shapelet-2015_10.0.1.post21...
lsst-astrometry-net-0.50.2...
lsst-meas-astrom-2015_10.0.5...
lsst-astrometry-net-data-0.10.0.post56...
lsst-ip-isr-2015_10.0.5...
lsst-ip-diffim-2015_10.0.1.post31...
lsst-coadd-chisquared-2015_10.0.1.post27...
lsst-skymap-2015_10.0.4.post1...
lsst-pipe-tasks-2015_10.0.13...
lsst-meas-modelfit-2015_10.0.2.post12...
lsst-mysqlpython-1.2.3.1.post1...
lsst-obs-lsstsim-2015_10.0.4...
lsst-obs-sdss-2015_10.0.5...
lsst-xpa-2.1.15.3...
lsst-display-ds9-2015_10.0.post27...
lsst-psfex-2015_10.0.1.post8...
lsst-meas-extensions-psfex-2015_10.0.2.post22...
lsst-apps-0.11.0.post73...
lsst-apr-1.3.3.3...
lsst-apr-util-1.3.4.3...
lsst-log4cxx-0.10.0.6...
lsst-log-2015_10.0.2.post9...
lsst-sqlalchemy-1.0.8.2.post1...
lsst-db-2015_10.0.post15...
lsst-cat-2015_10.0.1.post16...
lsst-activemqcpp-3.9.0.2.post1...
lsst-ctrl-events-2015_10.0.2.post10...
lsst-ctrl-provenance-2015_10.0.1.post16...
lsst-ctrl-orca-2015_10.0.2.post17...
lsst-ctrl-execute-2015_10.0.3.post17...
lsst-ctrl-platform-gordon-0.11.0.post44...
lsst-ctrl-platform-lsst-0.11.0.post44...
lsst-mariadb-0.10.1.9...
lsst-scisql-0.3.4.1.post1...
lsst-datarel-2015_10.0.1.post41...
lsst-distrib-0.11.0.post86...
lsst-pyephem-3.7.5.1.1...
lsst-palpy-1.7.0...
lsst-freetds-0.91.112.2...
lsst-pymssql-2.1.1.post3...
setuptools-git-1.1...
lsst-pykg-config-1.2.0.post5...
lsst-healpy-1.8.1.post15...
lsst-sims-data-0.10.1...
lsst-sims-utils-2.1.2.1.post1...
lsst-sims-catalogs-generation-2.1.2.1.post1...
lsst-sims-sed-library-2014.10.06...
lsst-sims-dustmaps-0.10.1.3...
lsst-throughputs-master.20160107215629...
lsst-sims-photutils-2.1.2.2.post1...
lsst-sims-coordutils-2.1.2.post8...
lsst-sims-maf-2.1.3.post3...
lsst-tmv-0.72.3...
lsst-galsim-1.2.2.1.post18...
lsst-sims-catalogs-measures-2.1.2.post4...
lsst-sims-catutils-2.1.2.2.post2...
lsst-sims-galsiminterface-2.1.2.post12...
lsst-sncosmo-1.1.1.1...
astropy-helpers-1.1.1...
lsst-sims-master.20150928224422.post102...
done. generating rebuild script: already built: eups-1.5.9-7 will build: lsst-doxygen-1.8.5.1-323f995_3 already built: lsst-product-configs-0.0.1-0 will build: lsst-sconsutils-2015_11.0.2.post1-9960fc2_0 will build: lsst-eups-environment-2015_10.0.post11-0 will build: lsst-boost-1.59.5-5 will build: lsst-base-2015_10.0.2-89e5401_0 will build: lsst-pex-exceptions-2015_10.0.1.post11-f7aee88_0 will build: lsst-utils-2015_10.0.3.post9-530cd83_0 will build: lsst-daf-base-2015_10.0.3.post9-4d32b56_0 will build: lsst-mysqlclient-5.1.73.3-3 will build: lsst-pex-logging-2015_10.0.1.post11-f6c85b3_0 will build: lsst-pex-policy-2015_10.0.1.post12-d4eb80a_0 will build: lsst-python-d2to1-0.2.12-4 will build: lsst-stsci-distutils-0.3.7.1-b22a065_3 will build: lsst-pyfits-3.2.4.2.post1-3 will build: lsst-daf-persistence-2015_10.0.6.post5-5a5f333_0 will build: lsst-pex-config-2015_10.0.1.post11-c006da1_0 will build: lsst-eigen-3.2.5-5 will build: lsst-fftw-3.3.4.2-3 will build: lsst-ndarray-0.10.1.post55-0 will build: lsst-cfitsio-3360.4-4 will build: lsst-wcslib-4.14.2-0 will build: lsst-minuit2-5.28.00.2-3 will build: lsst-gsl-1.16.3-3 will build: lsst-afw-2015_10.0.14-7c5ed66_0 will build: lsst-geom-0.10.0.post47-0 will build: lsst-skypix-0.10.0.post330-0 will build: lsst-daf-butlerutils-2015_10.0.4.post1-ca2b07e_0 will build: lsst-obs-test-2015_10.0.3.post6-c4c7f1b_0 will build: lsst-pipe-base-2015_10.0.5-417e839_0 will build: lsst-coadd-utils-2015_10.0.4.post11-74d0a18_0 will build: lsst-meas-base-2015_10.0.4.post11-87d2c31_0 will build: lsst-meas-algorithms-2015_10.0.4.post3-270941b_0 will build: lsst-meas-deblender-2015_10.0.3.post22-a5a97e7_0 will build: lsst-shapelet-2015_10.0.1.post21-7c5d349_0 will build: lsst-astrometry-net-0.50.2-0 will build: lsst-meas-astrom-2015_10.0.5-7977c95_0 will build: lsst-astrometry-net-data-0.10.0.post56-0 will build: lsst-ip-isr-2015_10.0.5-1b37747_0 will build: lsst-ip-diffim-2015_10.0.1.post31-c9560c6_0 will build: lsst-coadd-chisquared-2015_10.0.1.post27-470bb2d_0 will build: lsst-skymap-2015_10.0.4.post1-65a3ccf_0 will build: lsst-pipe-tasks-2015_10.0.13-4f8857f_0 will build: lsst-meas-modelfit-2015_10.0.2.post12-4183c33_0 will build: lsst-mysqlpython-1.2.3.1.post1-3 will build: lsst-obs-lsstsim-2015_10.0.4-17d213c_0 will build: lsst-obs-sdss-2015_10.0.5-b0be3e2_0 will build: lsst-xpa-2.1.15.3-0 will build: lsst-display-ds9-2015_10.0.post27-0 will build: lsst-psfex-2015_10.0.1.post8-eb1eb67_0 will build: lsst-meas-extensions-psfex-2015_10.0.2.post22-ce306f0_0 will build: lsst-apps-0.11.0.post73-0 will build: lsst-apr-1.3.3.3-0 will build: lsst-apr-util-1.3.4.3-0 will build: lsst-log4cxx-0.10.0.6-0 will build: lsst-log-2015_10.0.2.post9-81df675_0 will build: lsst-sqlalchemy-1.0.8.2.post1-0 will build: lsst-db-2015_10.0.post15-0 will build: lsst-cat-2015_10.0.1.post16-a13b24a_0 will build: lsst-activemqcpp-3.9.0.2.post1-0 will build: lsst-ctrl-events-2015_10.0.2.post10-b6a79c0_0 will build: lsst-ctrl-provenance-2015_10.0.1.post16-ef242a4_0 will build: lsst-ctrl-orca-2015_10.0.2.post17-4f54dab_0 will build: lsst-ctrl-execute-2015_10.0.3.post17-4ff3670_0 will build: lsst-ctrl-platform-gordon-0.11.0.post44-0 will build: lsst-ctrl-platform-lsst-0.11.0.post44-0 will build: lsst-mariadb-0.10.1.9-0 will build: lsst-scisql-0.3.4.1.post1-0 will build: lsst-datarel-2015_10.0.1.post41-63a66a4_0 will build: lsst-distrib-0.11.0.post86-0 will build: lsst-pyephem-3.7.5.1.1-7 will build: lsst-palpy-1.7.0-7 will build: lsst-freetds-0.91.112.2-4 already built: setuptools-git-1.1-py27_0 will build: lsst-pymssql-2.1.1.post3-4 will build: lsst-pykg-config-1.2.0.post5-7 will build: lsst-healpy-1.8.1.post15-3 will build: lsst-sims-data-0.10.1-0 will build: lsst-sims-utils-2.1.2.1.post1-fd59bd9_0 will build: lsst-sims-catalogs-generation-2.1.2.1.post1-ced3926_0 will build: lsst-sims-sed-library-2014.10.06-0 will build: lsst-sims-dustmaps-0.10.1.3-ffe99e1_3 will build: lsst-throughputs-master.20160107215629-3df0d76b78_2 will build: lsst-sims-photutils-2.1.2.2.post1-ecf83be_0 will build: lsst-sims-coordutils-2.1.2.post8-0 will build: lsst-sims-maf-2.1.3.post3-0 will build: lsst-tmv-0.72.3-0 will build: lsst-galsim-1.2.2.1.post18-6f8a81a_0 will build: lsst-sims-catalogs-measures-2.1.2.post4-0 will build: lsst-sims-catutils-2.1.2.2.post2-9e89ac4_0 will build: lsst-sims-galsiminterface-2.1.2.post12-0 already built: astropy-helpers-1.1.1-0 will build: lsst-sncosmo-1.1.1.1-cea177f_0 will build: lsst-sims-master.20150928224422.post102-c2d9827e8b_0 done. calling /home/cohen/lsst/dm/conda-lsst/recipes/rebuild.sh to build: lsst-doxygen (ver 1.8.5.1-323f995_3) [from doxygen-1.8.5-1-g323f995] ... ERROR Traceback (most recent call last): File "/home/cohen/lsst/dm/conda-lsst/bin/conda-lsst", line 149, in args.func(config, args) File "/home/cohen/lsst/dm/conda-lsst/bin/conda-lsst", line 30, in main_make_recipes subprocess.check_call('bash %s/rebuild.sh' % (config.output_dir), shell=True) File "/home/cohen/lsst/dm/conda-lsst/miniconda/lib/python2.7/subprocess.py", line 540, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'bash /home/cohen/lsst/dm/conda-lsst/recipes/rebuild.sh' returned non-zero exit status 255 cohen@port-jct:~/lsst/dm/conda-lsst$

johannct commented 8 years ago

the log is pretty atrocious, but only the last line, already know, seems relevant.... _build.txt

mjuric commented 8 years ago

You're missing flex. From the log file:

  Checking for perl... using /usr/bin/perl
  Checking for flex... not found!
johannct commented 8 years ago

darn, thanks and sorry! I missed it in the log....

johannct commented 8 years ago

for the record, I was also missing bison.

johannct commented 8 years ago

I am kipping this issue open for the different troubles I have, as it might prove useful to others trying conda install on their laptop (esp. Ubuntu 14.04 as mine). So I am now failing at lsst-utils, where numpy seems to have a problem at compilation. The only place I could see a hint at a problem is in :

Checking who built the CC compiler...error: no result CC is unknown version unknown Checking for C++11 support Checking whether the C++ compiler worksyes C++11 supported with '-std=c++11' Setting up environment to build package 'utils'. Ignoring prefix /home/cohen/lsst/dm/conda-lsst/miniconda/envs/_build/var/opt/eups/Linux64/utils/2015_10.0-3-g530cd83+9 from EUPS_PATH numpy was found but did not pass configuration checks.

I attach the log file again. Sorry for the trouble! _build.txt

johannct commented 8 years ago

I went a little farther down in debugging by adding a message printing in /home/cohen/lsst/dm/conda-lsst/miniconda/envs/_build/opt/lsst/lsst-product-configs/ups/numpy.cfg : def configure(self, conf, packages, check=False, build=True): lsst.sconsUtils.log.info("Configuring package '%s'." % self.name) try: import numpy except ImportError,msg: print msg #here return False conf.env.AppendUnique(XCPPPATH=numpy.get_include().split()) return True which I think should be added in general. This now tells me that the problem is that libgfortran.so.1 is not found. Indeed I have libgfortran.so.3 in my Linux Ubuntu 14.04 box, so I tried a soft link. But then numpy importing bombs again : /home/cohen/lsst/dm/conda-lsst/miniconda/envs/_build/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so: undefined symbol: _gfortran_copy_string So it looks like I need to rebuild numpy in miniconda. Is it possible, or has it been downloaded prebuilt? In the latter case, I guess I can replace the miniconda numpy by my system native one.... but that is a nice loophole for incompatibilities down the line....