Support custom pip command

[reubano]

I have my python3 versions of pip and pip-accel installed as pip3 and pip-accel3 respectively. However, when I run pip-accel3, it installs packages with pip (the python2 version). Is there a way to specify which pip version to use?

[xolox]

I don't know what to make of the issue you reported so I guess I'm missing some essential context.

When pip-accel invokes pip it does so at the Python API level, that is to say it imports from the pip namespace. The Python import statement doesn't support versioning; you'll just get the first module available on sys.path. This means pip-accel doesn't control what version of pip it will import.

Are you talking about a system wide installation or virtual environments? Because it kind of sounds like your Python installation is broken; Python 2 modules shouldn't be available for import in a Python 3 interpreter.

[reubano]

I'm referring to a system wide installation. I have py2 and py3. When you run pip, what you get back depends on how $PATH is defined. Since my py2 install takes priority, pip is the py2 version. I smylinked py3's pip --> pip3. Also, this is all on the command line... I'm not importing modules.

pip

reubano@tokpro [~]$ pip --version
pip 7.1.2 from /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (python 2.7)
reubano@tokpro [~]$ ls /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pip
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pip
reubano@tokpro [~]$ pip3 --version
pip 7.1.2 from /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip-7.1.2-py3.5.egg (python 3.5)
reubano@tokpro [~]$ ls /opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/pip
/opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/pip

pip-accel

reubano@tokpro [~]$ l `which pip-accel`
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/pip-accel
reubano@tokpro [~]$ l `which pip-accel3`
/Users/reubano/bin/pip-accel3 -> /opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/pip-accel

So my assumption is that the pip that pip-accel3 runs references the py2 version. When I pip-accel3 <package> on a package that uses future to provide py2 & 3 compatibility, I see that future is downloaded as well. This doesn't happen if I pip3 <package>.

[xolox]

Two questions:

1. Does anything break during or after you run the pip-accel3 install <package> command or are you worried that something might be going wrong (even though everything works fine at first glance)?
2. Can you give me the actual pip3 install <package> and pip-accel3 install <package> commands so I can try to reproduce the behavior you are describing?

---

A bit more context about how I understand the situation thus far, because it may enable you to follow my train of thought so we can get to the bottom of what's going on here:

1. You created a symbolic link /Users/reubano/bin/pip-accel3 which points to /opt/local/Library/Frameworks/Python.framework/Versions/3.5/bin/pip-accel.
2. When you run the pip-accel3 command it will start a Python 3.5 interpreter to load the pip_accel.* modules which import the pip.* modules.
3. Because we started with a Python 3.5 interpreter the Python 2.7 module search path will be irrelevant, so pip-accel should definitely be importing the version of pip installed alongside pip-accel under /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages.
4. The version of pip-accel installed in the Python 3.5 environment should never even encounter the version of pip installed in the Python 2.7 environment because pip-accel never runs pip as a subprocess, it just imports the pip.* modules.

[xolox]

A small addition: I can think of other reasons why the future package might be downloaded on Python 3.x, but to confirm my suspicions I'll need to take a look at the actual Python packages involved. For example wheel distributions support conditional dependencies, e.g. to mark a dependency as Python 2.x only. Source distributions require other handling to implement this same logic, so if pip prefers a wheel distribution but pip-accel for some reason prefers the source distribution, you could be seeing different (though possibly harmless) behavior.

[reubano]

1. Me seeing future in the download led me to believe something might be going wrong.
2. However, I was just trying to reproduce the behavior with pygogo and I see pip3 is now downloading future as well :(. I can't remember if I did something differently the first time around that avoided that particular download.

FYI, here is the optional py2 logic:

setup.py

...
if sys.version_info.major == 2:
    requirements.append('future==0.15.2')

[reubano]

Working now! I figured out the proper way to declare optional dependencies and tested it out on a new repo. In case anyone else needs to do the same:

setup.py

...
# Conditional sdist dependencies:
if 'bdist_wheel' not in sys.argv and sys.version_info.major == 2:
    requirements = py2_requirements
else:
    requirements = py3_requirements

# Conditional bdist_wheel dependencies:
extras_require = sorted(set(py2_requirements).difference(py3_requirements))

setup(
   ...
    install_requires=requirements,
    extras_require={':python_version<"3.0"': extras_require},
    ...)

Thanks for the help!!

[xolox]

Hi Reuben,

Glad to hear you got things working and thanks for following up and documenting the solution you found!