Having Compile Errors with pip install

[xcthulhu]

I am currently installing pyds9 via pip, which is a dependency for zachopy

As of ~30 minutes ago, ds9 throws a compile error when installed this way:

Attached is the error output for my machine, running MacOSX 10.10 with python 2.7.11 installed via homebrew:

Best match: pyds9 1.8.1
Processing pyds9.git
Writing /var/folders/fd/z1w7mk8n3tlc2slqmwhp8ngh0000gn/T/easy_install-xVBT5T/pyds9.git/setup.cfg
Running setup.py -q bdist_egg --dist-dir /var/folders/fd/z1w7mk8n3tlc2slqmwhp8ngh0000gn/T/easy_install-xVBT5T/pyds9.git/egg-dist-tmp-E4rmh2
/Users/mpwd/Documents/TESS/SPyFFI/test/venv/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.
  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')
Initializing astropy_helpers submodule with: `git submodule update --init -- astropy_helpers`
warning: no files found matching 'COPYING'
warning: no files found matching '*.c' under directory '*.pyx'
warning: no files found matching '*.pxd' under directory '*.pyx'
warning: no files found matching '*' under directory 'scripts'
no previously-included directories found matching 'build'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/api'
warning: no previously-included files matching '*.o' found anywhere in distribution
pre_build_ext_hook command hook build_ext raised an exception: %s

Traceback (most recent call last):
  File "/private/var/folders/fd/z1w7mk8n3tlc2slqmwhp8ngh0000gn/T/easy_install-xVBT5T/pyds9.git/astropy_helpers/astropy_helpers/setup_helpers.py", line 289, in run_command_hooks
    hook_obj(cmd_obj)
  File "pyds9/setup_package.py", line 91, in pre_build_ext_hook
    sp.check_call([os.path.join('.', 'configure'), '--without-tcl'])
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 535, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 522, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

error: Setup script exited with 1

[montefra]

@xcthulhu : how did you installed it? Can you give us the exact command?

It seems to me that you have cloned the repo and installed using python setup.py install or pip install .. It doesn't work because it uses a submodule (xpa) that is not initialized. For now you should initialize and update the submodules:

git submodule init
git submodule update

And then rerun the install.

A quick search shows that there have been discussion to add generic support for submodules and also some PR. I'll investigate and see what I can do.

Alternatively you can install the pyds9 directly from the git repo with:

pip install git+https://github.com/ericmandel/pyds9.git

or clone the git repository recursively with

git clone --recursive https://github.com/ericmandel/pyds9.git

[xcthulhu]

I am trying to set up ds9 (as well as zachopy) to be installable via pip.

For now, I am making a fork of this project and registering pyds9 with pypi

This name is rightfully yours, but I'm afraid the build system you have set up will not work for pypi. When you have it fixed this I will delete my module I have registered with pypi and you can register pyds9 yourself.

[montefra]

I am trying to set up ds9 (as well as zachopy) to be installable via pip. For now, I am making a fork of this project and registering pyds9 with pypi

I can install pyds9 with pip. Do you mean install it easily without having the repository or doing pip install git+...?

This name is rightfully yours, but I'm afraid the build system you have set up will not work for pypi.

Why not? What is the problem?

I tested installing the current pyds9 master from the gituhub repo, my local copy and the tar ball created by python setup.py sdist and I had no issue. The only problem is that cextern/xpa submodule is not initialized and updated automatically as it happens with astropy-helpers. I've found some issue about this but I don't know the status. I plan to investigate.

Also @ericmandel tested the installation on a Mac and it did work.

When you have it fixed this I will delete my module I have registered with pypi and you can register pyds9 yourself.

Ok. We'll let you know

[xcthulhu]

I can install pyds9 with pip. Do you mean install it easily without having the repository or doing pip install git+...?

I am trying to get my project to automatically all of its dependencies with pip install ... including pyds9. The xpa thing is a blocker.

For now I have a fork of the 1.8.1 release with a few minor changes to make setup.py work with pypi

The only problem is that cextern/xpa submodule is not initialized and updated automatically as it happens with astropy-helpers.

You guys should probably consider just vendoring xpa (like you did for the v1.8.1 release) or publishing a separate pypi module for it.

For now, the fact that I can't get this thing to compile is a non-issue for me.

[montefra]

Now I get it:

• If you do pip install git+..... it works: for some reason pip clone the repository recursively, so the submodules are expanded and the installation works.
• If you clone the repository yourself and run python setup.py sdist or something like this to create and/or upload tar files or wheels, only astropy_helpers is initialised, but no other package. There has been some movement about this last year astropy/astropy-helpers#155, but has being abandoned.
• If you clone the repository recursively all it's fine.

So I think that until that PR or something similar is implemented, we'll have to do with a local copy of xpa. I'll do it in the next couple of days (@ericmandel has any reserve).

[ericmandel]

@montefra No objections to using a local copy of xpa ... I always considered submodules to be slightly magical ...

[montefra]

@ericmandel: of ericmandel/xpa should I use the tag 2.1.17 or the current head of the trunk? Are you planning any new tag any time soon?

[ericmandel]

@montefra I'm not planning any work on xpa at all (for the past few years, we've only made a small number of changes to fix compiler warnings). So you should do whatever is easiest ...

[montefra]

@ericmandel: ok.

Then I'll consider xpa as stable. For now I add the current trunk into pyds9. I'll try to check from time to time if there are updates.

[xcthulhu]

This build is probably not going to work with pypi.

You should get rid of all of these submodules.

Instead of having astropy-helpers as a submodule, you might consider passing to the install_requires keyword that the setup function takes in setup.py. Here is an example from a library I am currently maintaining:

https://github.com/TESScience/SPyFFI/blob/master/setup.py#L36

@ericmandel: It would be nice for upstream consumers of xpa if you published to pypi as well, but I know you are busy and this sort of thing is pretty annoying.

[montefra]

This build is probably not going to work with pypi.

You should get rid of all of these submodules.

In PR #41 I have replaced the xpa submodule with a copy of the master

Instead of having astropy-helpers as a submodule, you might consider passing to the install_requires keyword that the setup function takes in setup.py. Here is an example from a library I am currently maintaining:

https://github.com/TESScience/SPyFFI/blob/master/setup.py#L36

I'm following the astropy instructions that suggests to add astropy-helpers as a submodule. ah_bootstrap.py takes care of getting astropy-helper.

When running any python setup.py command on a clean clone of pyds9, ah_bootstrap.py does a

Initializing astropy_helpers submodule with: `git submodule update --init -- astropy_helpers`

@ericmandel: It would be nice for upstream consumers of xpa if you published to pypi as well, but I know you are busy and this sort of thing is pretty annoying.

xpa is a pure c code. I don't think will work on pypi.

[xcthulhu]

When running any python setup.py...

I've never run that command. I find running setup.py directly to be unreliable when testing deployment.

(In particular, setuptools uses dependency_links which is not supported by pip - see here for a discussion.)

To test your deployment, I would run:

# cd /tmp
# virtualenv venv
# ./venv/bin/pip install git+https://github.com/montefra/pyds9.git@xpa_submodule

I just ran this and everything worked, I'll go sign off on PR #41.

[montefra]

When running any python setup.py... I've never run that command. I find running setup.py directly to be unreliable when testing deployment.

I usually just run python setup.py sdist upload ... to build a source distribution (basically a a tar ball) and upload to a local pypi-like server on other projects and it works fine.

(In particular, setuptools uses dependency_links which is not supported by pip - see here for a discussion.)

I know. It's annoying. That's why I ended up setting up a local pypi to distribute some code between colleagues.

To test your deployment, I would run:

cd /tmp virtualenv venv ./venv/bin/pip install git+https://github.com/montefra/pyds9.git@xpa_submodule I just ran this and everything worked, I'll go sign off on PR #41.

Great. Thank you for testing. I successfully tested installing from that branch repo and from the tar create by the sdist command on linux under python 2 and 3.

[montefra]

I've merged #41 back into trunk. Thank again.