Having trouble with automated build-all.py on osx Jenkins

jchodera commented 9 years ago

The build dies partway through trying to build msmbuilder:

conda build --output msmbuilder
Traceback (most recent call last):
  File "build-all.py", line 24, in <module>
    build('msmbuilder')
  File "build-all.py", line 14, in build
    shell('conda build %s' % name)
  File "build-all.py", line 5, in <lambda>
    shell = lambda cmd: check_output(cmd, shell=True)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx/miniconda/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'conda build msmbuilder' returned non-zero exit status 1

Unfortunately, there's not much more helpful information.

Full output is here

Any ideas?

kyleabeauchamp commented 9 years ago

Looks like an mdtraj version requirement issue in the msmbuilder recipe. The capi function requires a pretty new version of mdtraj.

kyleabeauchamp commented 9 years ago

I think #66 might help

jchodera commented 9 years ago

Still failing, but potentially new error:

Error: No packages found in current osx-64 channels matching: munkres
Building conda packages on grid of python / numpy versions
This will take a while... (15 minutes or more)

Full output: https://jenkins.choderalab.org/job/conda-omnia-release-osx/31/console

rmcgibbo commented 9 years ago

https://github.com/omnia-md/conda-recipes/tree/master/munkres

On Tue, Dec 30, 2014 at 5:57 PM, John Chodera notifications@github.com wrote:

Still failing, but potentially new error:

Error: No packages found in current osx-64 channels matching: munkres Building conda packages on grid of python / numpy versions This will take a while... (15 minutes or more)

Full output: https://jenkins.choderalab.org/job/conda-omnia-release-osx/31/console

— Reply to this email directly or view it on GitHub https://github.com/omnia-md/conda-recipes/issues/64#issuecomment-68408078 .

jchodera commented 9 years ago

I see that it's there, but I'm unclear about why this is causing a failure.

conda build will first try to grab the package from binstar if it isn't installed, and if that fails, it will try to build it locally if a recipe exists in the local subdirectory. munkres is listed in the msmbuilder meta.yaml as only being needed for testing---that's correct, yes?

rmcgibbo commented 9 years ago

Yes, that's correct. It's used as a reference implementation of some functionality for testing.

jchodera commented 9 years ago

Maybe conda doesn't automatically build testing dependencies, but only build dependencies?

rmcgibbo commented 9 years ago

conda build will first try to grab the package from binstar if it isn't installed, and if that fails...

It's already at https://binstar.org/omnia/munkres. Does your build environment have that channel added?

jchodera commented 9 years ago

No, the build environment is only using the jenkins channel right now to avoid inconsistencies.

Could it be that munkres is not listed anywhere in the requirements: section? The documentation for meta.yaml seems to suggest that only things listed there are automatically installed.

jchodera commented 9 years ago

No, wait, that doesn't make sense. It's clearly trying to reach out and install it.

rmcgibbo commented 9 years ago

You could just build it an push it to your jenkins channel. IIRC, it's just a pure python package.

jchodera commented 9 years ago

Trying to automate this. build-all.py should presumably be something we want to build everything, without manual intervention, right?

rmcgibbo commented 9 years ago

Sure. You're call. I'm on vacation :)

jchodera commented 9 years ago

So am I.

jchodera commented 9 years ago

I've switched to using the other conda-build-all script:

cd conda-recipes
./conda-build-all ./*

The automated osx build actually succeeds on Jenkins! https://jenkins.choderalab.org/job/conda-omnia-release-osx-2/3/console

Here's the complete list of files uploaded to the binstar jenkins channel: https://binstar.org/organization/omnia/dashboard/channels/jenkins

The next step would be to get this to work inside the osx VM on Jenkins, but this is at least a big step forward!

jchodera commented 9 years ago

Whoops, looks like I had temporarily disabled the binstar uploads. Just a moment for that to rebuild.

jchodera commented 9 years ago

OK, all packages that successfully built for osx have now been uploaded (manually): https://binstar.org/organization/omnia/dashboard/channels/jenkins

The next run (takes ~1.5h) should get them automatically.

However, there are a number of funny things in the console output during the build: https://jenkins.choderalab.org/job/conda-omnia-release-osx-2/6/console

Here's what I noticed:

The build for packmol fails because build.sh tries to use /usr/bin/gfortran (@kyleabeauchamp):

BUILD START: packmol-1.0.0-0
Fetching package metadata: ....
Solving package specifications: .+ export 'CFLAGS=-I/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/include -arch x86_64'
+ CFLAGS='-I/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/include -arch x86_64'
+ export 'LDFLAGS=-L/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/lib -arch x86_64'
+ LDFLAGS='-L/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/lib -arch x86_64'
+ make
 ------------------------------------------------------ 
 Compiling packmol with /usr/bin/gfortran 
 Flags: -O3 -ffast-math  
 ------------------------------------------------------ 
make: /usr/bin/gfortran: Permission denied
make: *** [serial] Error 1

The following NEW packages will be INSTALLED:

    zlib: 1.2.7-1

Linking packages ...
 |                                                                       |   0%
[zlib-1.2.7-1 /Users/jenkins-slave/workspace/conda-omnia-release] |######| 100%
[      COMPLETE      ] |#################################################| 100%
Removing old work directory
Source cache directory is: /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/src_cache
Downloading source to cache: packmol.tar.gz
packmol.tar.gz   0% |                              | ETA:  --:--:--   0.00  B/s
packmol.tar.gz  15% |####                           | ETA:  0:00:00  16.93 MB/s
packmol.tar.gz  31% |#########                      | ETA:  0:00:00  24.98 MB/s
packmol.tar.gz  47% |##############                 | ETA:  0:00:00  29.71 MB/s
packmol.tar.gz  63% |###################            | ETA:  0:00:00  32.96 MB/s
packmol.tar.gz  79% |########################       | ETA:  0:00:00  35.37 MB/s
packmol.tar.gz  95% |#############################  | ETA:  0:00:00  37.12 MB/s
packmol.tar.gz 100% |###############################| ETA:  0:00:00  34.66 MB/s
packmol.tar.gz 100% |###############################| Time: 0:00:00  31.35 MB/s
Extracting download
Package: packmol-1.0.0-0
source tree in: /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/work/packmol
Command failed: /bin/bash -x -e /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/conda-recipes/packmol/build.sh

The build for pdbfixer fails because the recipe does not specify numpy in the requirements:

BUILD START: pdbfixer-1.1.dev0-py27_0
Fetching package metadata: ....
Solving package specifications: .+ /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/bin/python setup.py install
Traceback (most recent call last):
  File "setup.py", line 9, in <module>
    import numpy
ImportError: No module named numpy

The following NEW packages will be INSTALLED:

    fftw3f:     3.3.3-1   
    openmm:     6.2-py27_0
    openssl:    1.0.1j-4  
    python:     2.7.9-1   
    readline:   6.2-2     
    setuptools: 7.0-py27_0
    sqlite:     3.8.4.1-0 
    tk:         8.5.15-0  
    zlib:       1.2.7-1   

Linking packages ...
 |                                                                       |   0%
[sqlite-3.8.4.1-0 /Users/jenkins-slave/workspace/conda-omnia-release] |  |  11%
[zlib-1.2.7-1 /Users/jenkins-slave/workspace/conda-omnia-release] |#     |  22%
[openssl-1.0.1j-4 /Users/jenkins-slave/workspace/conda-omnia-release] |  |  33%
[tk-8.5.15-0 /Users/jenkins-slave/workspace/conda-omnia-release] |###    |  44%
[readline-6.2-2 /Users/jenkins-slave/workspace/conda-omnia-release] |##  |  55%
[fftw3f-3.3.3-1 /Users/jenkins-slave/workspace/conda-omnia-release] |##  |  66%
[python-2.7.9-1 /Users/jenkins-slave/workspace/conda-omnia-release] |### |  77%
[openmm-6.2-py27_0 /Users/jenkins-slave/workspace/conda-omnia-release] | |  88%
[setuptools-7.0-py27_0 /Users/jenkins-slave/workspace/conda-omnia-release] || 100%
[      COMPLETE      ] |#################################################| 100%
Removing old work directory
Source cache directory is: /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/src_cache
Found source in cache: master.zip
Extracting download
Package: pdbfixer-1.1.dev0-py27_0
source tree in: /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/work/openmmtools-master
Command failed: /bin/bash -x -e /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/conda-recipes/pdbfixer/build.sh

The build for pyhmc fails due to some issue with the source code retrieval (@rmcgibbo):

BUILD START: pyhmc-0.1.1-np18py27_0
Fetching package metadata: ....
Solving package specifications: .Cloning into bare repository '/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/git_cache/rmcgibbo_pyhmc.git'...
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
An unexpected error has occurred, please consider sending the
following traceback to the conda GitHub issue tracker at:

    https://github.com/conda/conda-build/issues

Include the output of the command 'conda info' in your report.

The following NEW packages will be INSTALLED:

    cython:     0.21.2-py27_0
    numpy:      1.8.2-py27_0 
    openssl:    1.0.1j-4     
    python:     2.7.9-1      
    readline:   6.2-2        
    setuptools: 7.0-py27_0   
    sqlite:     3.8.4.1-0    
    tk:         8.5.15-0     
    zlib:       1.2.7-1      

Linking packages ...
 |                                                                       |   0%
[readline-6.2-2 /Users/jenkins-slave/workspace/conda-omnia-release] |    |  11%
[sqlite-3.8.4.1-0 /Users/jenkins-slave/workspace/conda-omnia-release] |  |  22%
[zlib-1.2.7-1 /Users/jenkins-slave/workspace/conda-omnia-release] |##    |  33%
[openssl-1.0.1j-4 /Users/jenkins-slave/workspace/conda-omnia-release] |  |  44%
[tk-8.5.15-0 /Users/jenkins-slave/workspace/conda-omnia-release] |###    |  55%
[python-2.7.9-1 /Users/jenkins-slave/workspace/conda-omnia-release] |##  |  66%
[cython-0.21.2-py27_0 /Users/jenkins-slave/workspace/conda-omnia-release] ||  77%
[numpy-1.8.2-py27_0 /Users/jenkins-slave/workspace/conda-omnia-release] ||  88%
[setuptools-7.0-py27_0 /Users/jenkins-slave/workspace/conda-omnia-release] || 100%
[      COMPLETE      ] |#################################################| 100%
Removing old work directory
Traceback (most recent call last):
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/bin/conda-build", line 5, in <module>
    sys.exit(main())
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 110, in main
    args_func(args, p)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 312, in args_func
    args.func(args, p)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/main_build.py", line 272, in execute
    build.build(m, verbose=not args.quiet, post=post)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/build.py", line 310, in build
    source.provide(m.path, m.get_section('source'))
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/source.py", line 236, in provide
    git_source(meta, recipe_dir)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/site-packages/conda_build/source.py", line 100, in git_source
    check_call([git, 'clone', '--mirror', git_url, cache_repo_arg], cwd=recipe_dir)
  File "/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/git', 'clone', '--mirror', u'git@github.com:rmcgibbo/pyhmc.git', u'/Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/git_cache/rmcgibbo_pyhmc.git']' returned non-zero exit status 128

The build for pymbar with python 3.x fails (@kyleabeauchamp):

BUILD START: pymbar-2.1beta0-np18py33_0
Fetching package metadata: ....
Solving package specifications: .+ /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/envs/_build/bin/python setup.py install
Traceback (most recent call last):
  File "setup.py", line 136, in <module>
    main()
  File "setup.py", line 132, in main
    setupKeywords = buildKeywordDictionary()
  File "setup.py", line 121, in buildKeywordDictionary
    for key in sorted( setupKeywords.iterkeys() ):
AttributeError: 'dict' object has no attribute 'iterkeys'

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    scipy-0.14.0               |       np18py33_0        10.6 MB

The following NEW packages will be INSTALLED:

    numpy:      1.8.2-py33_0     
    openssl:    1.0.1j-4         
    python:     3.3.5-3          
    readline:   6.2-2            
    scipy:      0.14.0-np18py33_0
    setuptools: 7.0-py33_0       
    sqlite:     3.8.4.1-0        
    tk:         8.5.15-0         
    xz:         5.0.5-0          
    zlib:       1.2.7-1          
...
Extracting packages ...
 |                                                                       |   0%
[scipy               ] |#################################################| 100%
[      COMPLETE      ] |#################################################| 100%
Linking packages ...
[      COMPLETE      ] |                                                 |   0%
[sqlite-3.8.4.1-0 /Users/jenkins-slave/workspace/conda-omnia-release] |  |  10%
[zlib-1.2.7-1 /Users/jenkins-slave/workspace/conda-omnia-release] |#     |  20%
[openssl-1.0.1j-4 /Users/jenkins-slave/workspace/conda-omnia-release] |  |  30%
[xz-5.0.5-0 /Users/jenkins-slave/workspace/conda-omnia-release] |###     |  40%
[tk-8.5.15-0 /Users/jenkins-slave/workspace/conda-omnia-release] |###    |  50%
[readline-6.2-2 /Users/jenkins-slave/workspace/conda-omnia-release] |##  |  60%
[python-3.3.5-3 /Users/jenkins-slave/workspace/conda-omnia-release] |##  |  70%
[numpy-1.8.2-py33_0 /Users/jenkins-slave/workspace/conda-omnia-release] ||  80%
[setuptools-7.0-py33_0 /Users/jenkins-slave/workspace/conda-omnia-release] ||  90%
[scipy-0.14.0-np18py33_0 /Users/jenkins-slave/workspace/conda-omnia-release] || 100%
[      COMPLETE      ] |#################################################| 100%
Removing old work directory
Source cache directory is: /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/src_cache
Found source in cache: v2.1.0-beta.tar.gz
Extracting download
Package: pymbar-2.1beta0-np18py33_0
source tree in: /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/miniconda/conda-bld/work/pymbar-2.1.0-beta
Command failed: /bin/bash -x -e /Users/jenkins-slave/workspace/conda-omnia-release-osx-2/conda-recipes/pymbar/build.sh

jchodera commented 9 years ago

I'll post these as separate issues.

jchodera commented 9 years ago

I'm going to close this since all remaining problems have separate issues addressing them.

omnia-md / conda-recipes

Having trouble with automated build-all.py on osx Jenkins #64