search

pantsbuild / pants

The Pants Build System
https://www.pantsbuild.org
Apache License 2.0
3.29k stars 628 forks source link

Investigate pants operation under the Anaconda 1.7.0 python distribution #1751

Closed jsirois closed 9 years ago

jsirois commented 9 years ago

As reported by laali to pants-devel here, and quoted below:

Hi,

I am new to pants. I want to use it to package a python script that depends on Anaconda 1.7 and PyQuante 1.6.4 so that users do not have to worry about installing them. However I have not yet been able to install pants on Anaconda 1.7. I downloaded pants from github (https://github.com/pantsbuild/pants.git) This is the output I get when I run ./pants

[: pants] ./pants Bootstrapping pants_deps with requirements: /home/laalitha/codes/pants/3rdparty/python/requirements.txt /home/laalitha/codes/pants/3rdparty/python/twitter/commons/requirements.txt /home/laalitha/codes/pants/pants-plugins/3rdparty/python/requirements.txt Using /home/laalitha/codes/anaconda/bin/python2.7 ~/codes/pants/build-support ~/codes/pants Downloading https://pypi.python.org/packages/source/v/virtualenv/virtualenv-12.1.1.tar.gz... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1641k 100 1641k 0 0 1334k 0 0:00:01 0:00:01 --:--:-- 1338k virtualenv-12.1.1/ virtualenv-12.1.1/AUTHORS.txt virtualenv-12.1.1/bin/ virtualenv-12.1.1/bin/rebuild-script.py virtualenv-12.1.1/docs/ virtualenv-12.1.1/docs/changes.rst virtualenv-12.1.1/docs/conf.py virtualenv-12.1.1/docs/development.rst virtualenv-12.1.1/docs/index.rst virtualenv-12.1.1/docs/installation.rst virtualenv-12.1.1/docs/make.bat virtualenv-12.1.1/docs/Makefile virtualenv-12.1.1/docs/reference.rst virtualenv-12.1.1/docs/userguide.rst virtualenv-12.1.1/LICENSE.txt virtualenv-12.1.1/MANIFEST.in virtualenv-12.1.1/PKG-INFO virtualenv-12.1.1/README.rst virtualenv-12.1.1/scripts/ virtualenv-12.1.1/scripts/virtualenv virtualenv-12.1.1/setup.cfg virtualenv-12.1.1/setup.py virtualenv-12.1.1/virtualenv.egg-info/ virtualenv-12.1.1/virtualenv.egg-info/dependency_links.txt virtualenv-12.1.1/virtualenv.egg-info/entry_points.txt virtualenv-12.1.1/virtualenv.egg-info/not-zip-safe virtualenv-12.1.1/virtualenv.egg-info/pbr.json virtualenv-12.1.1/virtualenv.egg-info/PKG-INFO virtualenv-12.1.1/virtualenv.egg-info/SOURCES.txt virtualenv-12.1.1/virtualenv.egg-info/top_level.txt virtualenv-12.1.1/virtualenv.py virtualenv-12.1.1/virtualenv_embedded/ virtualenv-12.1.1/virtualenv_embedded/activate.bat virtualenv-12.1.1/virtualenv_embedded/activate.csh virtualenv-12.1.1/virtualenv_embedded/activate.fish virtualenv-12.1.1/virtualenv_embedded/activate.ps1 virtualenv-12.1.1/virtualenv_embedded/activate.sh virtualenv-12.1.1/virtualenv_embedded/activate_this.py virtualenv-12.1.1/virtualenv_embedded/deactivate.bat virtualenv-12.1.1/virtualenv_embedded/distutils-init.py virtualenv-12.1.1/virtualenv_embedded/distutils.cfg virtualenv-12.1.1/virtualenv_embedded/site.py virtualenv-12.1.1/virtualenv_support/ virtualenv-12.1.1/virtualenv_support/init.py virtualenv-12.1.1/virtualenv_support/pip-6.1.1-py2.py3-none-any.whl virtualenv-12.1.1/virtualenv_support/setuptools-15.0-py2.py3-none-any.whl ‘virtualenv-12.1.1’ -> ‘virtualenv.dist’ ~/codes/pants New python executable in /home/laalitha/codes/pants/build-support/pants_dev_deps.venv/bin/python2.7 Also creating executable in /home/laalitha/codes/pants/build-support/pants_dev_deps.venv/bin/python ERROR: The executable /home/laalitha/codes/pants/build-support/pants_dev_deps.venv/bin/python2.7 is not functioning ERROR: It thinks sys.prefix is u'/usr' (should be u'/home/laalitha/codes/pants/build-support/pants_dev_deps.venv') ERROR: virtualenv is not compatible with this system or executable

Failed to create venv.

Details of the git

[: pants] git branch

  • master [: pants] git log -1 commit ca8d338e11a2c5ec5685e2a6a61218755a7a5b2b Author: Cody Gibb codyhgibb@gmail.com Date: Fri Jun 26 12:42:11 2015 -0700

    Documented how to enable caching for tasks.

    Reviewed at https://rbcommons.com/s/twitter/r/2420/

I also did ./pants --prefix=/home/laali/codes/anaconda/ it also gave the same error. I would appreciate any advice to make this work.

Thanks

jsirois commented 9 years ago

I was a bit terrified to run the Anaconda installer so I setup an ubuntu box to do this in:

$ cat Vagrantfile
Vagrant.configure(2) do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.provider "virtualbox" do |vb|
    vb.gui = false
    vb.memory = "1024"
  end
  config.vm.provision "shell", inline: <<-SHELL
    sudo apt-get update
    sudo apt-get upgrade -y
    sudo apt-get install -y git
  SHELL
end

And used the current 2.2.0 Anaconda installer:

$ vagrant up
$ vagrant ssh
vagrant@vagrant-ubuntu-trusty-64:~/pants$ export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
vagrant@vagrant-ubuntu-trusty-64:~/pants$ curl -O https://3230d63b5fc54e62148e-c95ac804525aac4b6dba79b00b39d1d3.ssl.cf1.rackcdn.com/Anaconda-2.2.0-Linux-x86_64.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  317M  100  317M    0     0  3024k      0  0:01:47  0:01:47 --:--:-- 3124k
vagrant@vagrant-ubuntu-trusty-64:~/pants$ bash Anaconda-2.2.0-Linux-x86_64.sh -bf
PREFIX=/home/vagrant/anaconda
installing: python-2.7.9-2 ...
installing: conda-3.10.0-py27_0 ...
...
installing: _cache-0.0-x0 ...
Python 2.7.9 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
vagrant@vagrant-ubuntu-trusty-64:~/pants$ python --version
Python 2.7.9 :: Anaconda 2.2.0 (64-bit)
vagrant@vagrant-ubuntu-trusty-64:~/pants$ python2 --version
Python 2.7.9 :: Anaconda 2.2.0 (64-bit)
vagrant@vagrant-ubuntu-trusty-64:~/pants$ python2.7 --version
Python 2.7.9 :: Anaconda 2.2.0 (64-bit)

Then to repro, although I hit no errors:

vagrant@vagrant-ubuntu-trusty-64:~$ git clone https://github.com/pantsbuild/pants.git
vagrant@vagrant-ubuntu-trusty-64:~/pants$ ./pants
...
Pants 0.0.33 https://pypi.python.org/pypi/pantsbuild.pants/0.0.33
...

So I have a few questions:

  1. Do you have a download URL for the exact version of Ananconda you use so I can attempt a more faithful repro?
  2. A tangent for a little later - do you need to clone the pants repo or would it suffice just to pip install pantsbuild.pants ? Maybe you're cloning pantsbuild/pants to build a pex for distribution to your users?

Thanks for any information you can provide.

laalitha commented 9 years ago

Sorry jsirois for the late reply. Thanks much for testing this issue.

  1. The exact Anacoda version I used was 1.7.0
  2. I cloned the github repo since it is the one that worked. The need for pantsbuild is to build a pex out of my script (acbn0.py) for distribution to users.

Now some info on my script acbn0.py It depends on numpy < 1.9, scipy==0.15.1, matplotlib=1.1.1 and PyQuante 1.6.4

So to get acbn0.py working on vanilla python I installed python 2.7.10 from source then installed virtualenv 13.1 from source and created a virtual environment. Then installed numpy==1.8.2 scipy==0.15.1 matplotlib==1.1.1 and PyQuante 1.6.4 to that virtual environment. The script works as desired in this environment. Now since everything is installed to my workspace I cloned pants and tried to build a PEX from the script. It only depends on one other script and I included it as a dependency. But pants is unable to find scipy,numpy and matplotlib libraries

My BUILD file is 

python_library(name="integs-lib",
                           sources=['integs.py',],
                          )
python_binary(
  name = 'acbn0',
  source = 'acbn0.py',
  dependencies= [':integs-lib',],
)

The output is 

./pants run acbn0_pex:acbn0
INFO] Detected git repository at /home/laalitha/codes/asupka/pants on branch master

11:10:48 00:00 [main]
               (To run a reporting server: ./pants server)
11:10:48 00:00   [bootstrap]
11:10:48 00:00   [setup]
11:10:48 00:00     [parse]
               Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> resolve -> compile -> resources -> binary -> run
11:10:49 00:01   [bootstrap]
11:10:49 00:01     [bootstrap-jvm-tools]
11:10:49 00:01   [imports]
11:10:49 00:01     [ivy-imports]
11:10:49 00:01   [unpack-jars]
11:10:49 00:01     [unpack-jars]
11:10:49 00:01   [deferred-sources]
11:10:49 00:01     [deferred-sources]
11:10:49 00:01   [gen]
11:10:49 00:01     [thrift]
11:10:49 00:01     [protoc]
11:10:49 00:01       [execute]
11:10:49 00:01     [antlr]
11:10:49 00:01     [ragel]
11:10:49 00:01     [jaxb]
11:10:49 00:01       [execute]
11:10:49 00:01     [wire]
11:10:49 00:01       [execute]
11:10:49 00:01     [aapt]
11:10:49 00:01     [scrooge]
11:10:49 00:01     [spindle]
11:10:49 00:01   [resolve]
11:10:49 00:01     [ivy]
11:10:49 00:01   [compile]
11:10:49 00:01     [compile]
11:10:49 00:01     [jvm]
11:10:49 00:01       [jvm-compilers]
11:10:49 00:01     [python-eval]
                   Invalidated 1 target.
11:10:49 00:01       [eval-targets].
                         Failed to eval acbn0_pex:acbn0
                         Failed to eval 'entry_point acbn0':
                           File "/tmp/tmpReGkIj/acbn0.py", line 7, in <module>
                             import scipy.io as sio
                         No module named scipy.io

FAILURE: Failed to eval acbn0_pex:acbn0

               Waiting for background workers to finish.
11:10:49 00:01   [complete]
               FAILURE
jsirois commented 9 years ago

Pants aims to do builds that include exactly the dependencies you declare and no more as a general principle. Your BUILD file examples only make pants aware of local sources and not the 3rdparty dependencies. The integs-lib target owns integs.py and has 0 dependencies and the acbn0 target owns acbn0.py and has a single internal dependency on the integs-lib target. No- where do you express dependencies on 3rdparty libraries like numpy or scipy - this is the missing bit. Pants ignores your local virtualenv contents intentionally, since those local bits are not reproduceable on someone elses machine (at least they are not reproducable in easy way by pants itself!).

So what you'll need is something more like this:

BUILD

python_requirements('requirements.txt')

python_library(
  name='integs-lib',
  sources=['integs.py'],
  dependencies=[
    ':matplotlib',
    ':numpy',
    ':PyQuante',
    ':scipy',
  ]
)

python_binary(
  name = 'acbn0',
  source = 'acbn0.py',
  dependencies= [':integs-lib'],
)

requirements.txt

matplotlib==1.1.1
numpy==1.8.2
PyQuante==1.6.4
scipy==0.15.1

There are a few ways to write down 3rd-party python dependencies in a BUILD file and this is just one. The more important thing to focus on here is the integs-lib library explicitly lists all its dependencies. Generally for each import in each python file owned by the integs-lib target, there should be a 3rd party dependency if the import is not satisfied by the python standard library or by other local python_library targets (source code you own).

You'll see examples of this in the pants codebase itself. We have 2 buckets of 3rdparty deps:

  1. twitter/commons 3rdparty deps

    https://github.com/pantsbuild/pants/blob/master/3rdparty/python/twitter/commons/requirements.txt https://github.com/pantsbuild/pants/blob/master/3rdparty/python/twitter/commons/BUILD

  2. All else:

    https://github.com/pantsbuild/pants/blob/master/3rdparty/python/requirements.txt https://github.com/pantsbuild/pants/blob/master/3rdparty/python/BUILD

And then one example python_library that has deps from both of these buckets, plus internal deps: https://github.com/pantsbuild/pants/blob/master/src/python/pants/reporting/BUILD#L4

laalitha commented 9 years ago

Thanks for that thorough explanation. It cleared up how to specify dependencies for me. My new BUILD file now looks like

python_requirements('requirements.txt')

python_library(name="integs-lib",
               sources=['integs.py',],
               dependencies=[
                ':numpy',
               ],
)
python_binary(
  name = 'acbn0',
  source = 'acbn0.py',
  dependencies= [
    ':integs-lib',
    ':ipython',
    ':numpy',
    ':scipy',
    ':matplotlib',
    ':PyQuante',
    ],
)

with a requirements.txt as

numpy==1.8.2
scipy==0.15.1
matplotlib==1.1.1
PyQuante==1.6.4
ipython==3.2.0

It now detects all 3rd-party python dependencies except PyQuante

./pants run acbn0_pex:acbn0INFO] Detected git repository at /home/laalitha/codes/asupka/pants on branch master

14:03:59 00:00 [main]
               (To run a reporting server: ./pants server)
14:03:59 00:00   [bootstrap]
14:03:59 00:00   [setup]
14:03:59 00:00     [parse]
               Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> resolve -> compile -> resources -> binary -> run
14:03:59 00:00   [bootstrap]
14:03:59 00:00     [bootstrap-jvm-tools]
14:03:59 00:00   [imports]
14:03:59 00:00     [ivy-imports]
14:03:59 00:00   [unpack-jars]
14:03:59 00:00     [unpack-jars]
14:03:59 00:00   [deferred-sources]
14:03:59 00:00     [deferred-sources]
14:03:59 00:00   [gen]
14:03:59 00:00     [thrift]
14:03:59 00:00     [protoc]
14:03:59 00:00       [execute]
14:03:59 00:00     [antlr]
14:03:59 00:00     [ragel]
14:03:59 00:00     [jaxb]
14:03:59 00:00       [execute]
14:03:59 00:00     [wire]
14:03:59 00:00       [execute]
14:03:59 00:00     [aapt]
14:03:59 00:00     [scrooge]
14:03:59 00:00     [spindle]
14:03:59 00:00   [resolve]
14:03:59 00:00     [ivy]
14:03:59 00:00   [compile]
14:03:59 00:00     [compile]
14:03:59 00:00     [jvm]
14:03:59 00:00       [jvm-compilers]
14:03:59 00:00     [python-eval]
                   Invalidated 1 target.
14:03:59 00:00       [eval-targets].Building chroot for [PythonBinary(BuildFileAddress(FilesystemBuildFile(/home/laalitha/codes/asupka/pants/acbn0_pex/BUILD), acbn0))]:
  Dumping library: PythonLibrary(BuildFileAddress(FilesystemBuildFile(/home/laalitha/codes/asupka/pants/acbn0_pex/BUILD), integs-lib))
  Dumping library: PythonBinary(BuildFileAddress(FilesystemBuildFile(/home/laalitha/codes/asupka/pants/acbn0_pex/BUILD), acbn0))
  Dumping requirement: numpy==1.8.2
  Dumping requirement: ipython==3.2.0
  Dumping requirement: scipy==0.15.1
  Dumping requirement: matplotlib==1.1.1
  Dumping requirement: PyQuante==1.6.4

               Waiting for background workers to finish.
14:04:00 00:01   [complete]
               FAILURE

Exception caught:
  File "/home/laalitha/codes/asupka/pants/src/python/pants/bin/pants_exe.py", line 86, in <module>
    main()
  File "/home/laalitha/codes/asupka/pants/src/python/pants/bin/pants_exe.py", line 81, in main
    _run(exiter)
  File "/home/laalitha/codes/asupka/pants/src/python/pants/bin/pants_exe.py", line 74, in _run
    result = goal_runner.run()
  File "/home/laalitha/codes/asupka/pants/src/python/pants/bin/goal_runner.py", line 212, in run
    result = self._do_run()
  File "/home/laalitha/codes/asupka/pants/src/python/pants/bin/goal_runner.py", line 269, in _do_run
    return engine.execute(context, self.goals)
  File "/home/laalitha/codes/asupka/pants/src/python/pants/engine/engine.py", line 26, in execute
    self.attempt(context, goals)
  File "/home/laalitha/codes/asupka/pants/src/python/pants/engine/round_engine.py", line 212, in attempt
    goal_executor.attempt(explain)
  File "/home/laalitha/codes/asupka/pants/src/python/pants/engine/round_engine.py", line 45, in attempt
    task.execute()
  File "/home/laalitha/codes/asupka/pants/src/python/pants/backend/python/tasks/python_eval.py", line 48, in execute
    compiled = self._compile_targets(invalidation_check.invalid_vts)
  File "/home/laalitha/codes/asupka/pants/src/python/pants/backend/python/tasks/python_eval.py", line 57, in _compile_targets
    return_code = self._compile_target(target)
  File "/home/laalitha/codes/asupka/pants/src/python/pants/backend/python/tasks/python_eval.py", line 138, in _compile_target
    executable_file_content=executable_file_content) as chroot:
  File "/home/laalitha/local/python2.7/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/laalitha/codes/asupka/pants/src/python/pants/backend/python/tasks/python_task.py", line 150, in temporary_chroot
    extra_requirements, executable_file_content)
  File "/home/laalitha/codes/asupka/pants/src/python/pants/backend/python/tasks/python_task.py", line 168, in _build_chroot
    chroot.dump()
  File "/home/laalitha/codes/asupka/pants/src/python/pants/backend/python/python_chroot.py", line 211, in dump
    distributions = self._resolve_multi(reqs_to_build, find_links)
  File "/home/laalitha/codes/asupka/pants/src/python/pants/backend/python/python_chroot.py", line 249, in _resolve_multi
    cache_ttl=self._python_setup.resolver_cache_ttl)
  File "/home/laalitha/codes/asupka/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 220, in resolve
    raise Unsatisfiable('Cannot satisfy requirements: %s' % requirement_set[requirement.key])

Exception message: Cannot satisfy requirements: [PythonRequirement(PyQuante==1.6.4)]

PyQuante 1.6.4 was installed from source into the virtual environment pip has a record of the same installation

pip show PyQuante
---
Metadata-Version: 1.1
Name: PyQuante
Version: 1.6.4
Summary: PyQuante: Quantum Chemistry in Python
Home-page: http://pyquante.sourceforge.net
Author: Rick Muller
Author-email: rmuller@sandia.gov
License: BSD
Location: /home/laalitha/codes/asupka/acbn0env/lib/python2.7/site-packages/PyQuante-1.6.4-py2.7-linux-x86_64.egg
Requires:

How do I make pants include it?

Thanks

jsirois commented 9 years ago

@patricklaw gave a response to your question about --find-links here: https://groups.google.com/d/topic/pants-devel/YItXgX1NKxk/discussion That response is part of the answer. PyQuante does not distribute on pypi (This is just a link back to their sourceforge site: https://pypi.python.org/pypi/PyQuante) and so a distribution must be built by-hand out of band from pants. Once done, the distribution can be put in a --find-links style directory that's either local or served up over http. With that directory in your [python-repos] repos list in pants.ini, resolution will now work.

At this point though you'll hit another issue (I tried this all out). Scipy has a setup.py that in-parts tests for numpy's presence and deals with its absence gracefully, but in others just assumes it exists. This leads to the following type of error when trying to install scipy first:

  1. Under pip:
$ virtualenv-2.7 /tmp/scipy
New python executable in /tmp/scipy/bin/python
Installing setuptools, pip...done.
$ source /tmp/scipy/bin/activate
(scipy)$ pip install scipy==0.15.1
Downloading/unpacking scipy==0.15.1
  Downloading scipy-0.15.1.tar.gz (11.4MB): 11.4MB downloaded
  Running setup.py (path:/tmp/scipy/build/scipy/setup.py) egg_info for package scipy
...
Downloading/unpacking numpy>=1.5.1 (from scipy==0.15.1)
  Downloading numpy-1.9.2.tar.gz (4.0MB): 4.0MB downloaded
  Running setup.py (path:/tmp/scipy/build/numpy/setup.py) egg_info for package numpy
    Running from numpy source directory.

    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '*.pyd' found anywhere in distribution
Installing collected packages: scipy, numpy
  Running setup.py install for scipy
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/scipy/build/scipy/setup.py", line 249, in <module>
        setup_package()
      File "/tmp/scipy/build/scipy/setup.py", line 237, in setup_package
        from numpy.distutils.core import setup
    ImportError: No module named numpy.distutils.core
  1. Under pex 1.0.1:
$ virtualenv-2.7 /tmp/pex
New python executable in /tmp/pex/bin/python
Installing setuptools, pip...done.
$ source /tmp/pex/bin/activate
(pex)$ pip install pex
Downloading/unpacking pex
  Downloading pex-1.0.1-py2.py3-none-any.whl (92kB): 92kB downloaded
Requirement already satisfied (use --upgrade to upgrade): setuptools>=2.2,<16 in /tmp/pex/lib/python2.7/site-packages (from pex)
Installing collected packages: pex
Successfully installed pex
Cleaning up...
(pex)jsirois@gill ~ $ which pex
/tmp/pex/bin/pex
(pex)jsirois@gill ~ $ pex scipy==0.15.1 -o scipy.pex
/home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
**** Failed to install scipy-0.15.1. stdout:

**** Failed to install scipy-0.15.1. stderr:
Traceback (most recent call last):
  File "<stdin>", line 7, in <module>
  File "setup.py", line 249, in <module>
    setup_package()
  File "setup.py", line 237, in setup_package
    from numpy.distutils.core import setup
ImportError: No module named numpy.distutils.core

Traceback (most recent call last):
  File "/home/jsirois/.pyenv/versions/pex/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/pex/bin/pex.py", line 509, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/pex/bin/pex.py", line 471, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/pex/resolver.py", line 177, in resolve
    distributions[package] = self.build(package, resolvable.options)
  File "/home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/pex/resolver.py", line 228, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/home/jsirois/.pyenv/versions/pex/lib/python2.7/site-packages/pex/resolver.py", line 147, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage(u'file:///home/jsirois/.pex/build/scipy-0.15.1.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

Currently pants can do no better than pex with poor setup.pys and it certainly can do no better than pip. The end result is you must pre-build some of the ill-behaved dependencies here and make them available to pants / pex / pip via --find-links flat dirs. I'll follow up shortly with my full working example including the minimal number of pre-builts required to get it working. The numpy ecosystem has never been fun to setup!

laalitha commented 9 years ago

Thanks @jsirois I really appreciate your help. Since I am using a slightly modified version of PyQuante I tried to make package and publish in PyPi. But as you said the setup.py is poorly written and therefore it did not work out. I will wait for your solution. Thanks again.

jsirois commented 9 years ago

OK - I've created a minimal example repository that has a self-contained setup that works for the case you describe: https://github.com/pantsbuild/issues-1751

Just follow the instructions in the README after cloning and let me know if these leaves any questions unanswered or problems remaining.

I'll close this issue assuming the example repo explains the missing bits satisfactorily - but feel free to re-open the issue if you run into further problems.

laalitha commented 9 years ago

Thanks @jsirois. Using the prebuilts of the minimal example I built the pex for my code. It was successful and it ran on while the path was set to the python 2.7.10 installation that I custom installed. But when I used the system python installation I got the following error

:scfU] python
Python 2.7.5 (default, Jun 24 2015, 00:41:19) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
: scfU] ./acbn0.pex input_dm_Ca.txt 
Traceback (most recent call last):
  File "/home/laalitha/codes/asupka/scfU/acbn0.pex/.bootstrap/_pex/pex.py", line 271, in execute
  File "/home/laalitha/codes/asupka/scfU/acbn0.pex/.bootstrap/_pex/pex.py", line 320, in execute_entry
  File "/home/laalitha/codes/asupka/scfU/acbn0.pex/.bootstrap/_pex/pex.py", line 337, in execute_module
  File "/usr/lib64/python2.7/runpy.py", line 180, in run_module
    fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "acbn0.py", line 7, in <module>
    import scipy.io as sio
  File "/home/laalitha/.pex/install/scipy-0.15.1-cp27-none-linux_x86_64.whl.fb43d471d239cf4cf736180eac47b8771e3a17c4/scipy-0.15.1-cp27-none-linux_x86_64.whl/scipy/io/__init__.py", line 85, in <module>
    from .matlab import loadmat, savemat, whosmat, byteordercodes
  File "/home/laalitha/.pex/install/scipy-0.15.1-cp27-none-linux_x86_64.whl.fb43d471d239cf4cf736180eac47b8771e3a17c4/scipy-0.15.1-cp27-none-linux_x86_64.whl/scipy/io/matlab/__init__.py", line 13, in <module>
    from .mio import loadmat, savemat, whosmat
  File "/home/laalitha/.pex/install/scipy-0.15.1-cp27-none-linux_x86_64.whl.fb43d471d239cf4cf736180eac47b8771e3a17c4/scipy-0.15.1-cp27-none-linux_x86_64.whl/scipy/io/matlab/mio.py", line 12, in <module>
    from .miobase import get_matfile_version, docfiller
  File "/home/laalitha/.pex/install/scipy-0.15.1-cp27-none-linux_x86_64.whl.fb43d471d239cf4cf736180eac47b8771e3a17c4/scipy-0.15.1-cp27-none-linux_x86_64.whl/scipy/io/matlab/miobase.py", line 22, in <module>
    from scipy.misc import doccer
  File "/home/laalitha/.pex/install/scipy-0.15.1-cp27-none-linux_x86_64.whl.fb43d471d239cf4cf736180eac47b8771e3a17c4/scipy-0.15.1-cp27-none-linux_x86_64.whl/scipy/misc/__init__.py", line 47, in <module>
    from scipy.special import comb, factorial, factorial2, factorialk
  File "/home/laalitha/.pex/install/scipy-0.15.1-cp27-none-linux_x86_64.whl.fb43d471d239cf4cf736180eac47b8771e3a17c4/scipy-0.15.1-cp27-none-linux_x86_64.whl/scipy/special/__init__.py", line 586, in <module>
    from ._ufuncs import *
ImportError: /home/laalitha/.pex/install/scipy-0.15.1-cp27-none-linux_x86_64.whl.fb43d471d239cf4cf736180eac47b8771e3a17c4/scipy-0.15.1-cp27-none-linux_x86_64.whl/scipy/special/_ufuncs.so: undefined symbol: PyUnicodeUCS2_DecodeUTF8

I also ran this on a completely different machine and I got 

laalitha]$ python
Python 2.7.6 |Anaconda 2.0.0 (64-bit)| (default, May 27 2014, 14:50:58) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io and https://binstar.org
>>> 
:laalitha]$ ./acbn0.pex 
Traceback (most recent call last):
  File "/tmp/laalitha/acbn0.pex/.bootstrap/_pex/pex.py", line 271, in execute
  File "/tmp/laalitha/acbn0.pex/.bootstrap/_pex/pex.py", line 320, in execute_entry
  File "/tmp/laalitha/acbn0.pex/.bootstrap/_pex/pex.py", line 337, in execute_module
  File "/home/laalitha/SW/anaconda/lib/python2.7/runpy.py", line 180, in run_module
    fname, loader, pkg_name)
  File "/home/laalitha/SW/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/laalitha/acbn0.pex/acbn0.py", line 7, in <module>
  File "/home/laalitha/.pex/install/scipy-0.15.1-cp27-none-linux_x86_64.whl.fb43d471d239cf4cf736180eac47b8771e3a17c4/scipy-0.15.1-cp27-none-linux_x86_64.whl/scipy/__init__.py", line 70, in <module>
    from numpy import show_config as show_numpy_config
  File "/home/laalitha/.pex/install/numpy-1.8.2-cp27-none-linux_x86_64.whl.c5689334978950e0a1fac02c605d3e72d6335439/numpy-1.8.2-cp27-none-linux_x86_64.whl/numpy/__init__.py", line 153, in <module>
    from . import add_newdocs
  File "/home/laalitha/.pex/install/numpy-1.8.2-cp27-none-linux_x86_64.whl.c5689334978950e0a1fac02c605d3e72d6335439/numpy-1.8.2-cp27-none-linux_x86_64.whl/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/laalitha/.pex/install/numpy-1.8.2-cp27-none-linux_x86_64.whl.c5689334978950e0a1fac02c605d3e72d6335439/numpy-1.8.2-cp27-none-linux_x86_64.whl/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/laalitha/.pex/install/numpy-1.8.2-cp27-none-linux_x86_64.whl.c5689334978950e0a1fac02c605d3e72d6335439/numpy-1.8.2-cp27-none-linux_x86_64.whl/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/laalitha/.pex/install/numpy-1.8.2-cp27-none-linux_x86_64.whl.c5689334978950e0a1fac02c605d3e72d6335439/numpy-1.8.2-cp27-none-linux_x86_64.whl/numpy/core/__init__.py", line 6, in <module>
    from . import multiarray
ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/laalitha/.pex/install/numpy-1.8.2-cp27-none-linux_x86_64.whl.c5689334978950e0a1fac02c605d3e72d6335439/numpy-1.8.2-cp27-none-linux_x86_64.whl/numpy/core/multiarray.so)

The one that worked was

scfU] python
Python 2.7.10 (default, Jul  6 2015, 15:24:48) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 
: scfU] ./acbn0.pex input_dm_Ca.txt 
openbabel not found in path, switching to PyQuante backend
libint extension not found, switching to normal ERI computation
.
.
.
.
jsirois commented 9 years ago

Yes - for libraries with native bits, its expected that the resulting pex will only run on (interpreters,platforms) it has built distributions for.

The 1st issue is generally not a problem; ie: a pex built on 2.7.6 on a given platform (say OSX), will work when run under python 2.7.10 also on OSX. Unfortunately you hit a problem described here where the two interpreters are not compatible at a lower level: http://stackoverflow.com/questions/11878666/python-undefined-symbol-pyunicodeucs2-decodeutf8

You might be able to work around this particular issue if and only if you have uniform machine environments, when this is implemented: https://github.com/pantsbuild/pants/issues/1768 At that point you can embed the full path to the specific interpreter the pex was built with since you know all machines will have the same (compatible) interpreter at that same path.

Looking at the second problem - pants allows you to build a pex that supports multiple platforms. The docs touch on this here: https://pantsbuild.github.io/howto_develop.html#building-a-pants-pex-for-production

You can do this globally for all your pants built pexes in a repo with the pants.ini [python-setup] platforms list option (defaults globally to ['current'], or in a python_binary directly like so:

python_binary(
  ...
  platforms=[       
   'current',       
   'linux-x86_64',      
   'macosx-10.4-x86_64',        
  ],
...
)

That platforms configuration says always build for osx and linux. This requires though that you have a bdist repo with eggs or wheels for each dependency you have 1-per platform. So you'd need to build OSX wheels and place them there and also build a set of linux wheels.

This doesn't magically work unless all your linux targets have the same libs. If you target, for example, both Cent OS 5 and 6, you'll hit issues at runtime with certain libs mismatching (If you built the platform specific wheels on Cent OS 6 they'll likelys fail on Cent OS 5).

laalitha commented 9 years ago

So I experimented on a few systems, and found out that if I built the pex on a system that has a older version of glibc library it will work on machines that have newer versions. But it will not work on machines with a glibc that is older than the one that was used to build the pex. I was trying to build it in a machine that had the oldest version I could find (2.12) but I am having trouble building the pex. Here's what I get:

issues-1751]$ PEX_VERBOSE=1 PANTS_VERBOSE=1 PYTHON_VERBOSE=1 ./pants -ldebug binary //:acbn0
DEBUG] Parsing BUILD file FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD).
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=numpy, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), numpy)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=scipy, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), scipy)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=matplotlib, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), matplotlib)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=PyQuante, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), PyQuante)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=ipython, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), ipython)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_library.PythonLibrary'>, name=integs-lib, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), integs-lib)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_binary.PythonBinary'>, name=acbn0, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), acbn0)
DEBUG] FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD) produced the following Addressables:
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), matplotlib): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=matplotlib, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), scipy): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=scipy, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), ipython): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=ipython, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), acbn0): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_binary.PythonBinary'>, name=acbn0, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), integs-lib): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_library.PythonLibrary'>, name=integs-lib, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), PyQuante): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=PyQuante, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), numpy): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=numpy, **kwargs=...)
DEBUG] Parsing BUILD file FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD.tools).
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.jvm.targets.jar_library.JarLibrary'>, name=nailgun-server, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD.tools), nailgun-server)
DEBUG] FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD.tools) produced the following Addressables:
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD.tools), nailgun-server): TargetAddressable(target_type=<class 'pants.backend.jvm.targets.jar_library.JarLibrary'>, name=nailgun-server, **kwargs=...)
DEBUG] Executing: git --git-dir=/home/laalitha/SW/issues-1751/.git --work-tree=/home/laalitha/SW/issues-1751 rev-parse --abbrev-ref HEAD
INFO] Detected git repository at /home/laalitha/SW/issues-1751 on branch master
DEBUG] Executing: git --git-dir=/home/laalitha/SW/issues-1751/.git --work-tree=/home/laalitha/SW/issues-1751 rev-parse HEAD
DEBUG] Executing: git --git-dir=/home/laalitha/SW/issues-1751/.git --work-tree=/home/laalitha/SW/issues-1751 describe --tags --always
DEBUG] Executing: git --git-dir=/home/laalitha/SW/issues-1751/.git --work-tree=/home/laalitha/SW/issues-1751 rev-parse --abbrev-ref HEAD

10:15:12 00:00 [main]
               (To run a reporting server: ./pants server)
10:15:12 00:00   [bootstrap]
10:15:12 00:00   [setup]
10:15:12 00:00     [parse]
               Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> resolve -> compile -> resources -> binary
10:15:12 00:00   [bootstrap]
10:15:12 00:00     [bootstrap-jvm-tools]DEBUG] Located Distribution(u'/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.35.x86_64/bin', minimum_version=None, maximum_version=None jdk=False) for constraints: minimum_version None, maximum_version None, jdk False

10:15:12 00:00   [imports]
10:15:12 00:00     [ivy-imports]
10:15:12 00:00   [unpack-jars]
10:15:12 00:00     [unpack-jars]
10:15:12 00:00   [deferred-sources]
10:15:12 00:00     [deferred-sources]
10:15:12 00:00   [gen]
10:15:12 00:00     [thrift]
10:15:12 00:00     [protoc]DEBUG] Selected protoc binary bootstrapped to: /home/laalitha/.cache/pants/bin/protobuf/linux/x86_64/2.4.1/protoc

10:15:12 00:00       [execute]
10:15:13 00:01     [antlr]
10:15:13 00:01     [ragel]
10:15:13 00:01     [jaxb]
10:15:13 00:01       [execute]
10:15:13 00:01     [wire]
10:15:13 00:01       [execute]
10:15:13 00:01   [resolve]
10:15:13 00:01     [ivy]DEBUG] Using previously resolved reports: [u'/home/laalitha/.ivy2/pants/internal-707bcedd7c7c17e4fe5d23c677367ea7b619bfa1-IvyResolveFingerprintStrategy_53c4bb9c6553-default.xml']

10:15:13 00:01   [compile]
10:15:13 00:01     [compile]
10:15:13 00:01     [jvm]
10:15:13 00:01       [jvm-compilers]
                     ::: created chunks(0)
10:15:13 00:01   [resources]
10:15:13 00:01     [prepare]
10:15:13 00:01     [services]
10:15:13 00:01   [binary]
10:15:13 00:01     [python-binary-create]
                       fetching https://pypi.python.org/packages/source/s/setuptools/setuptools-5.4.1.zip#md5=96bd961ab481c78825a5be8546f42a66
                       installing /tmp/tmpZowwxB/setuptools-5.4.1.zip
               Waiting for background workers to finish.
10:15:16 00:04   [complete]
               FAILURE

Exception message: [Errno 2] No such file or directory
laalitha commented 9 years ago

Also could the pex be built with static linking to GLIBC? Would that solve the GLIBC dependency?

laalitha commented 9 years ago

I also saw that the GLIBC version dependency could be avoided by using the compiler flag -fno-stack-protector. http://gbgames.com/blog/2008/04/linux-game-development-glibc_24-errors-solved/ How and where to include these compiler flag so that the pex will be version independent of GLIBC ?

jsirois commented 9 years ago

For your 1st question - can you provide more details? These would help:

  1. What OS are you trying and failing on - what python version?:

    I was trying to build it in a machine that had the oldest version I could find (2.12)

    Oldest version of what? I'm assuming you mean Linux.

  2. Please re-run with the failing command with --print-exception-stacktrace, *_VERBOSE env vars set to 5 instead of 1, and make sure to look for extra stdout/stderr output in the ./pants server web console log for the run.

I'll leave your last 2 questions to you to solve. Since this example includes a manual (though scripted) step to create prebuilt wheels, you could modify that step to produce statically linked wheels or wheels built with special compiler flags when you figure out how to do these things. I do see this advice for passing flags: http://stackoverflow.com/questions/6928110/how-may-i-override-the-compiler-gcc-flags-that-setup-py-uses-by-default

laalitha commented 9 years ago

What OS are you trying and failing on - what python version?:

OS: CentOS 6.6 libc version: 2.12 Python version: 2.7.10

Oldest version of what?

Oldest glibc version

re-run with the failing command with --print-exception-stacktrace, *_VERBOSE env vars set to 5

This gives me

    issues-1751]$ PEX_VERBOSE=5 PANTS_VERBOSE=5 PYTHON_VERBOSE=5 ./pants -ldebug --print-exception-stacktrace binary //:acbn0
DEBUG] Parsing BUILD file FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD).
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=numpy, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), numpy)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=scipy, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), scipy)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=matplotlib, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), matplotlib)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=PyQuante, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), PyQuante)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=ipython, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), ipython)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_library.PythonLibrary'>, name=integs-lib, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), integs-lib)
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.python.targets.python_binary.PythonBinary'>, name=acbn0, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), acbn0)
DEBUG] FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD) produced the following Addressables:
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), matplotlib): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=matplotlib, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), scipy): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=scipy, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), ipython): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=ipython, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), acbn0): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_binary.PythonBinary'>, name=acbn0, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), integs-lib): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_library.PythonLibrary'>, name=integs-lib, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), PyQuante): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=PyQuante, **kwargs=...)
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD), numpy): TargetAddressable(target_type=<class 'pants.backend.python.targets.python_requirement_library.PythonRequirementLibrary'>, name=numpy, **kwargs=...)
DEBUG] Parsing BUILD file FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD.tools).
DEBUG] Adding TargetAddressable(target_type=<class 'pants.backend.jvm.targets.jar_library.JarLibrary'>, name=nailgun-server, **kwargs=...) to the BuildFileParser address map with BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD.tools), nailgun-server)
DEBUG] FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD.tools) produced the following Addressables:
DEBUG]   * BuildFileAddress(FilesystemBuildFile(/home/laalitha/SW/issues-1751/BUILD.tools), nailgun-server): TargetAddressable(target_type=<class 'pants.backend.jvm.targets.jar_library.JarLibrary'>, name=nailgun-server, **kwargs=...)
DEBUG] Executing: git --git-dir=/home/laalitha/SW/issues-1751/.git --work-tree=/home/laalitha/SW/issues-1751 rev-parse --abbrev-ref HEAD
INFO] Detected git repository at /home/laalitha/SW/issues-1751 on branch master
DEBUG] Executing: git --git-dir=/home/laalitha/SW/issues-1751/.git --work-tree=/home/laalitha/SW/issues-1751 rev-parse HEAD
DEBUG] Executing: git --git-dir=/home/laalitha/SW/issues-1751/.git --work-tree=/home/laalitha/SW/issues-1751 describe --tags --always
DEBUG] Executing: git --git-dir=/home/laalitha/SW/issues-1751/.git --work-tree=/home/laalitha/SW/issues-1751 rev-parse --abbrev-ref HEAD

15:40:31 00:00 [main]
               (To run a reporting server: ./pants server)
15:40:31 00:00   [bootstrap]
15:40:31 00:00   [setup]
15:40:31 00:00     [parse]
               Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> resolve -> compile -> resources -> binary
15:40:31 00:00   [bootstrap]
15:40:31 00:00     [bootstrap-jvm-tools]DEBUG] Located Distribution(u'/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.35.x86_64/bin', minimum_version=None, maximum_version=None jdk=False) for constraints: minimum_version None, maximum_version None, jdk False

15:40:31 00:00   [imports]
15:40:31 00:00     [ivy-imports]
15:40:31 00:00   [unpack-jars]
15:40:31 00:00     [unpack-jars]
15:40:31 00:00   [deferred-sources]
15:40:31 00:00     [deferred-sources]
15:40:31 00:00   [gen]
15:40:31 00:00     [thrift]
15:40:31 00:00     [protoc]DEBUG] Selected protoc binary bootstrapped to: /home/laalitha/.cache/pants/bin/protobuf/linux/x86_64/2.4.1/protoc

15:40:31 00:00       [execute]
15:40:31 00:00     [antlr]
15:40:31 00:00     [ragel]
15:40:31 00:00     [jaxb]
15:40:31 00:00       [execute]
15:40:31 00:00     [wire]
15:40:31 00:00       [execute]
15:40:31 00:00   [resolve]
15:40:31 00:00     [ivy]DEBUG] Using previously resolved reports: [u'/home/laalitha/.ivy2/pants/internal-707bcedd7c7c17e4fe5d23c677367ea7b619bfa1-IvyResolveFingerprintStrategy_53c4bb9c6553-default.xml']

15:40:31 00:00   [compile]
15:40:31 00:00     [compile]
15:40:31 00:00     [jvm]
15:40:31 00:00       [jvm-compilers]
                     ::: created chunks(0)
15:40:31 00:00   [resources]
15:40:31 00:00     [prepare]
15:40:32 00:01     [services]
15:40:32 00:01   [binary]
15:40:32 00:01     [python-binary-create]
                       fetching https://pypi.python.org/packages/source/s/setuptools/setuptools-5.4.1.zip#md5=96bd961ab481c78825a5be8546f42a66pex: Fetching https://pypi.python.org/packages/source/s/setuptools/setuptools-5.4.1.zip#md5=96pex: Validated setuptools-5.4.1.zip (md5=96bd961ab481c78825a5be8546f42a66)                                                  
pex: Validated setuptools-5.4.1.zip (md5=96bd961ab481c78825a5be8546f42a66)
pex: Fetching https://pypi.python.org/packages/source/s/setuptools/setuptools-5.4.1.zip#md5=96bd961ab481c78825a5be8546f42a66: 287.8ms

                       installing /tmp/tmp7WIgUb/setuptools-5.4.1.zippex: Installing /tmp/tmpxSxQ44
               Waiting for background workers to finish.
15:40:32 00:01   [complete]
               FAILURE

Exception caught:
  File "/home/laalitha/.pants.d/bootstrap/pantsbuild.pants-latest/bin/pants", line 9, in <module>
    load_entry_point('pantsbuild.pants==0.0.34', 'console_scripts', 'pants')()
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 81, in main
    _run(exiter)
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 74, in _run
    result = goal_runner.run()
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 212, in run
    result = self._do_run()
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 269, in _do_run
    return engine.execute(context, self.goals)
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/engine/engine.py", line 26, in execute
    self.attempt(context, goals)
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/engine/round_engine.py", line 212, in attempt
    goal_executor.attempt(explain)
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/engine/round_engine.py", line 45, in attempt
    task.execute()
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/backend/python/tasks/python_binary_create.py", line 38, in execute
    self.create_binary(binary)
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/backend/python/tasks/python_binary_create.py", line 41, in create_binary
    interpreter = self.select_interpreter_for_targets(binary.closure())
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/backend/python/tasks/python_task.py", line 65, in select_interpreter_for_targets
    allowed_interpreters = OrderedSet(self.interpreter_cache.interpreters)
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/backend/python/tasks/python_task.py", line 51, in interpreter_cache
    self._interpreter_cache.setup(filters=self._compatibilities)
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/backend/python/interpreter_cache.py", line 129, in setup
    self._setup_cached(filters)
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/backend/python/interpreter_cache.py", line 87, in _setup_cached
    pi = self._interpreter_from_path(path, filters)
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/backend/python/interpreter_cache.py", line 73, in _interpreter_from_path
    return self._resolve(interpreter)
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/backend/python/interpreter_cache.py", line 144, in _resolve
    self._python_setup.setuptools_requirement())
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/backend/python/interpreter_cache.py", line 172, in _resolve_interpreter
    installer_provider)
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pants/backend/python/interpreter_cache.py", line 196, in _resolve_and_link
    dist_location = installer.bdist()
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pex/installer.py", line 25, in function_wrapper
    self._installed = self.run()
  File "/home/laalitha/.pants.d/bootstrap/pants.3xiZeJ/install/lib/python2.7/site-packages/pex/installer.py", line 107, in run
    cwd=self._source_dir)
  File "/home/laalitha/local/python/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/home/laalitha/local/python/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception

   Exception message: [Errno 2] No such file or directory

I could not see any errors reported in web console log. I am trying out the compiler flags. I will post the outcome

jsirois commented 9 years ago

Hmm, I'll try setting up a vagrant box in the https://github.com/pantsbuild/issues-1751 repository to repro your issue. I may need you to finish off the setup to match your environment exactly though.

laalitha commented 9 years ago

I tried to pass compiler flags when building scipy as CFLAGS=-static CXXFLAGS=-static python setupegg.py bdist_wheel but the compile was unsuccessful with

 gcc -pthread -shared -static build/temp.linux-x86_64-2.7/scipy/cluster/_hierarchy.o -Lbuild/temp.linux-x86_64-2.7 -o build/lib.linux-x86_64-2.7/scipy/cluster/_hierarchy.so
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/4.8.3/crtbeginT.o: relocation R_X86_64_32 against `__TMC_END__' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-redhat-linux/4.8.3/crtbeginT.o: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/4.8.3/crtbeginT.o: relocation R_X86_64_32 against `__TMC_END__' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-redhat-linux/4.8.3/crtbeginT.o: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
error: Command "gcc -pthread -shared -static build/temp.linux-x86_64-2.7/scipy/cluster/_hierarchy.o -Lbuild/temp.linux-x86_64-2.7 -o build/lib.linux-x86_64-2.7/scipy/cluster/_hierarchy.so" failed with exit status 1

I tried with CFLAGS="-fPIC -static" CXXFLAGS="-fPIC -static" python setupegg.py bdist_wheel but I still got the error for scipy

gcc -pthread -shared -fPIC -static build/temp.linux-x86_64-2.7/scipy/cluster/_hierarchy.o -Lbuild/temp.linux-x86_64-2.7 -o build/lib.linux-x86_64-2.7/scipy/cluster/_hierarchy.so
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/4.8.3/crtbeginT.o: relocation R_X86_64_32 against `__TMC_END__' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-redhat-linux/4.8.3/crtbeginT.o: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/4.8.3/crtbeginT.o: relocation R_X86_64_32 against `__TMC_END__' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-redhat-linux/4.8.3/crtbeginT.o: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
error: Command "gcc -pthread -shared -fPIC -static build/temp.linux-x86_64-2.7/scipy/cluster/_hierarchy.o -Lbuild/temp.linux-x86_64-2.7 -o build/lib.linux-x86_64-2.7/scipy/cluster/_hierarchy.so" failed with exit status 1
jsirois commented 9 years ago

If you can repro your failures with the vagrant box I've setup I can help you further - but I'll need to be able to have that to make reasonable progress. The vagrant box is here: https://github.com/pantsbuild/issues-1751/tree/master/prebuilts/centos/6.6/x86_64 When I follow the README I get a successful build and run of the test.pex on the centos box. To form your repro case you may need to setup another vagrant box (ie: works on CentOS 6.6, fails on CentOS 5.x) - but IIUC you first just need to tweak the Vagrantfile to provision to the state you need (ie: old glibc, etc). Let me know if you have any questions using this to setup a portable repro case and let me know when you get that repro case all setup.

laalitha commented 9 years ago

Thanks @jsirois but I could not reproduce the error in vagrant box. We gave up the building a pex and copied all the routines we were using from PyQuante to our script, since building a universal pex turned out to be problamatic. Thanks for all your help.

jsirois commented 9 years ago

OK. That's too bad, but I'll mark this closed.

laalitha commented 9 years ago

This was a good learning experience for me. Introduced me to the whole python packaging methods mainly pants, setuptools and vagrant boxes which was really cool. Thanks again for spending time on these issues.