project toml broke editable install with pip 19.1

[GavinHuttley]

Original report by GavinH (Bitbucket: 557058:e40c23e1-e273-4527-a2f8-5de5876e870d, ).

---

When you have pip 19.1 installed, I get error messages that indicates cogent3 cannot be installed in editable mode and use --no-use-pep517 but I cannot get that to work

Please see if you can reproduce and then fix!

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

(c3dev-test) tl@pc:~/pycogent3/PyCogent3Apps$ pip uninstall cogent3
Uninstalling cogent3-3.0a2:
  Would remove:
    /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages/cogent3-3.0a2.dist-info/*
    /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages/cogent3/*
Proceed (y/n)? y
  Successfully uninstalled cogent3-3.0a2
(c3dev-test) tl@pc:~/pycogent3/PyCogent3Apps$ pip uninstall numpy
Uninstalling numpy-1.16.3:
  Would remove:
    /home/tl/miniconda3/envs/c3dev-test/bin/f2py
    /home/tl/miniconda3/envs/c3dev-test/bin/f2py3
    /home/tl/miniconda3/envs/c3dev-test/bin/f2py3.7
    /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages/numpy-1.16.3.dist-info/*
    /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages/numpy/*
Proceed (y/n)? y
  Successfully uninstalled numpy-1.16.3
(c3dev-test) tl@pc:~/pycogent3/PyCogent3Apps$ pip --version
pip 19.1.1 from /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages/pip (python 3.7)
(c3dev-test) tl@pc:~/pycogent3/PyCogent3Apps$ pip install -e .
Obtaining file:///home/tl/pycogent3/PyCogent3Apps
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting numpy (from cogent3==3.0a2)
  Using cached https://files.pythonhosted.org/packages/bb/76/24e9f32c78e6f6fb26cf2596b428f393bf015b63459468119f282f70a7fd/numpy-1.16.3-cp37-cp37m-manylinux1_x86_64.whl
Requirement already satisfied: pandas in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from cogent3==3.0a2) (0.24.2)
Requirement already satisfied: plotly in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from cogent3==3.0a2) (3.9.0)
Requirement already satisfied: python-dateutil>=2.5.0 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from pandas->cogent3==3.0a2) (2.8.0)
Requirement already satisfied: pytz>=2011k in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from pandas->cogent3==3.0a2) (2019.1)
Requirement already satisfied: six in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from plotly->cogent3==3.0a2) (1.12.0)
Requirement already satisfied: nbformat>=4.2 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from plotly->cogent3==3.0a2) (4.4.0)
Requirement already satisfied: requests in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from plotly->cogent3==3.0a2) (2.22.0)
Requirement already satisfied: retrying>=1.3.3 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from plotly->cogent3==3.0a2) (1.3.3)
Requirement already satisfied: decorator>=4.0.6 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from plotly->cogent3==3.0a2) (4.4.0)
Requirement already satisfied: ipython_genutils in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from nbformat>=4.2->plotly->cogent3==3.0a2) (0.2.0)
Requirement already satisfied: traitlets>=4.1 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from nbformat>=4.2->plotly->cogent3==3.0a2) (4.3.2)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from nbformat>=4.2->plotly->cogent3==3.0a2) (3.0.1)
Requirement already satisfied: jupyter_core in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from nbformat>=4.2->plotly->cogent3==3.0a2) (4.4.0)
Requirement already satisfied: certifi>=2017.4.17 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from requests->plotly->cogent3==3.0a2) (2019.3.9)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from requests->plotly->cogent3==3.0a2) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from requests->plotly->cogent3==3.0a2) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from requests->plotly->cogent3==3.0a2) (1.24.3)
Requirement already satisfied: attrs>=17.4.0 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2->plotly->cogent3==3.0a2) (19.1.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2->plotly->cogent3==3.0a2) (0.15.2)
Requirement already satisfied: setuptools in /home/tl/miniconda3/envs/c3dev-test/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2->plotly->cogent3==3.0a2) (41.0.1)
Installing collected packages: numpy, cogent3
  Running setup.py develop for cogent3
Successfully installed cogent3 numpy-1.16.3

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

For some reason, I couldn’t replicate the issue: I uninstalled cogent3 and numpy. My pip version is 19.1.1 (may try upgrade to that version on your end?) Then I sucessfully installed. I did remember reading something about an older pip version (possibly 19.1) having this sort of problem, so that could be the cause of this bug. Let me know how it goes.

[GavinHuttley]

Original comment by GavinH (Bitbucket: 557058:e40c23e1-e273-4527-a2f8-5de5876e870d, ).

---

direct installation via pip worked for me too, but install via conda did not.

‌

$ conda env remove -n c3test
$ conda env create -n filename.yml

fails at the last step

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

Thats supposed to be an

-f

switch right?

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

I ran the following and successfully installed the conda environment.

conda env create -f test.yml

Can I see the output logs please?

[GavinHuttley]

Original comment by GavinH (Bitbucket: 557058:e40c23e1-e273-4527-a2f8-5de5876e870d, ).

---

post test.yml here please

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

name: c3dev-test
channels:
  - conda-forge
  - defaults
  - plotly
dependencies:
  - matplotlib
  - numpy
  - pip
  - python=3.7
  - plotly
  - plotly-orca
  - pip:
    - pandas
    - pytest
    - tqdm
    - ./PyCogent3Apps/ 

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

Was the ticket regarding the original yaml with all dev tools?

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

Actually, I think I’ve got it replicated. Is this what your output looks like?

Collecting package metadata: done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.6.8
  latest version: 4.6.14

Please update conda by running

    $ conda update -n base -c defaults conda

Downloading and Extracting Packages
terminado-0.8.2      | 23 KB     | ##################################### | 100% 
nodejs-11.14.0       | 16.6 MB   | ##################################### | 100% 
jupyter_client-5.2.4 | 63 KB     | ##################################### | 100% 
jupyterlab-0.35.6    | 10.6 MB   | ##################################### | 100% 
ipykernel-5.1.1      | 156 KB    | ##################################### | 100% 
pyzmq-18.0.1         | 491 KB    | ##################################### | 100% 
parso-0.4.0          | 64 KB     | ##################################### | 100% 
pexpect-4.7.0        | 78 KB     | ##################################### | 100% 
nb_conda_kernels-2.2 | 32 KB     | ##################################### | 100% 
pandoc-2.7.2         | 21.7 MB   | ##################################### | 100% 
notebook-5.7.8       | 7.2 MB    | ##################################### | 100% 
openmpi-3.1.4        | 4.0 MB    | ##################################### | 100% 
ipywidgets-7.4.2     | 92 KB     | ##################################### | 100% 
testpath-0.4.2       | 85 KB     | ##################################### | 100% 
zeromq-4.3.1         | 648 KB    | ##################################### | 100% 
pygments-2.4.0       | 660 KB    | ##################################### | 100% 
defusedxml-0.5.0     | 20 KB     | ##################################### | 100% 
widgetsnbextension-3 | 1.7 MB    | ##################################### | 100% 
ptyprocess-0.6.0     | 15 KB     | ##################################### | 100% 
jinja2-2.10.1        | 91 KB     | ##################################### | 100% 
nbconvert-5.5.0      | 375 KB    | ##################################### | 100% 
psutil-5.6.2         | 320 KB    | ##################################### | 100% 
ipython-7.5.0        | 1.1 MB    | ##################################### | 100% 
scipy-1.3.0          | 18.8 MB   | ##################################### | 100% 
mpi4py-3.0.1         | 643 KB    | ##################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: - Enabling nb_conda_kernels...
Status: enabled

/ Enabling notebook extension jupyter-js-widgets/extension...
      - Validating: OK

done

CondaEnvException: Pip subcommand failed with 
output: Obtaining file:///home/tl/pycogent3/PyCogent3Apps (from -r /home/tl/pycogent3/condaenv.57qe8phq.requirements.txt (line 12))

error: ++ /home/tl/miniconda3/bin/conda shell.posix hook
+ eval 'export CONDA_EXE="/home/tl/miniconda3/bin/conda"
# Copyright (C) 2012 Anaconda, Inc
# SPDX-License-Identifier: BSD-3-Clause

__conda_hashr() {
    if [ -n "${ZSH_VERSION:+x}" ]; then
        \rehash
    elif [ -n "${POSH_VERSION:+x}" ]; then
        :  # pass
    else
        \hash -r
    fi
}

__conda_activate() {
    if [ -n "${CONDA_PS1_BACKUP:+x}" ]; then
        # Handle transition from shell activated with conda <= 4.3 to a subsequent activation
        # after conda updated to >= 4.4. See issue #6173.
        PS1="$CONDA_PS1_BACKUP"
        \unset CONDA_PS1_BACKUP
    fi

    \local cmd="$1"
    shift
    \local ask_conda
    ask_conda="$(PS1="$PS1" "$CONDA_EXE" shell.posix "$cmd" "$@")" || \return $?
    \eval "$ask_conda"
    __conda_hashr
}

__conda_reactivate() {
    \local ask_conda
    ask_conda="$(PS1="$PS1" "$CONDA_EXE" shell.posix reactivate)" || \return $?
    \eval "$ask_conda"
    __conda_hashr
}

conda() {
    if [ "$#" -lt 1 ]; then
        "$CONDA_EXE"
    else
        \local cmd="$1"
        shift
        case "$cmd" in
            activate|deactivate)
                __conda_activate "$cmd" "$@"
                ;;
            install|update|upgrade|remove|uninstall)
                "$CONDA_EXE" "$cmd" "$@" && __conda_reactivate
                ;;
            *) "$CONDA_EXE" "$cmd" "$@" ;;
        esac
    fi
}

if [ -z "${CONDA_SHLVL+x}" ]; then
    \export CONDA_SHLVL=0
    PATH="$(dirname "$(dirname "$CONDA_EXE")")/condabin:${PATH:-}"
    \export PATH

    # We'\''re not allowing PS1 to be unbound. It must at least be set.
    # However, we'\''re not exporting it, which can cause problems when starting a second shell
    # via a first shell (i.e. starting zsh from bash).
    if [ -z "${PS1+x}" ]; then
        PS1=
    fi
fi

conda activate base'
++ export CONDA_EXE=/home/tl/miniconda3/bin/conda
++ CONDA_EXE=/home/tl/miniconda3/bin/conda
++ '[' -z x ']'
++ conda activate base
++ '[' 2 -lt 1 ']'
++ local cmd=activate
++ shift
++ case "$cmd" in
++ __conda_activate activate base
++ '[' -n '' ']'
++ local cmd=activate
++ shift
++ local ask_conda
+++ PS1=
+++ /home/tl/miniconda3/bin/conda shell.posix activate base
++ ask_conda='PS1='\''(base) '\''
\export CONDA_DEFAULT_ENV='\''base'\''
\export CONDA_EXE='\''/home/tl/miniconda3/bin/conda'\''
\export CONDA_PREFIX='\''/home/tl/miniconda3'\''
\export CONDA_PROMPT_MODIFIER='\''(base) '\''
\export CONDA_PYTHON_EXE='\''/home/tl/miniconda3/bin/python'\''
\export CONDA_SHLVL='\''1'\''
\export PATH='\''/home/tl/miniconda3/bin:/home/tl/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'\'''
++ eval 'PS1='\''(base) '\''
\export CONDA_DEFAULT_ENV='\''base'\''
\export CONDA_EXE='\''/home/tl/miniconda3/bin/conda'\''
\export CONDA_PREFIX='\''/home/tl/miniconda3'\''
\export CONDA_PROMPT_MODIFIER='\''(base) '\''
\export CONDA_PYTHON_EXE='\''/home/tl/miniconda3/bin/python'\''
\export CONDA_SHLVL='\''1'\''
\export PATH='\''/home/tl/miniconda3/bin:/home/tl/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'\'''
+++ PS1='(base) '
+++ export CONDA_DEFAULT_ENV=base
+++ CONDA_DEFAULT_ENV=base
+++ export CONDA_EXE=/home/tl/miniconda3/bin/conda
+++ CONDA_EXE=/home/tl/miniconda3/bin/conda
+++ export CONDA_PREFIX=/home/tl/miniconda3
+++ CONDA_PREFIX=/home/tl/miniconda3
+++ export 'CONDA_PROMPT_MODIFIER=(base) '
+++ CONDA_PROMPT_MODIFIER='(base) '
+++ export CONDA_PYTHON_EXE=/home/tl/miniconda3/bin/python
+++ CONDA_PYTHON_EXE=/home/tl/miniconda3/bin/python
+++ export CONDA_SHLVL=1
+++ CONDA_SHLVL=1
+++ export PATH=/home/tl/miniconda3/bin:/home/tl/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+++ PATH=/home/tl/miniconda3/bin:/home/tl/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
++ __conda_hashr
++ '[' -n '' ']'
++ '[' -n '' ']'
++ hash -r
+ conda activate /home/tl/miniconda3/envs/c3dev-test-dev
+ '[' 2 -lt 1 ']'
+ local cmd=activate
+ shift
+ case "$cmd" in
+ __conda_activate activate /home/tl/miniconda3/envs/c3dev-test-dev
+ '[' -n '' ']'
+ local cmd=activate
+ shift
+ local ask_conda
++ PS1='(base) '
++ /home/tl/miniconda3/bin/conda shell.posix activate /home/tl/miniconda3/envs/c3dev-test-dev
+ ask_conda='PS1='\''(c3dev-test-dev) '\''
\export CONDA_DEFAULT_ENV='\''c3dev-test-dev'\''
\export CONDA_PREFIX='\''/home/tl/miniconda3/envs/c3dev-test-dev'\''
\export CONDA_PREFIX_1='\''/home/tl/miniconda3'\''
\export CONDA_PROMPT_MODIFIER='\''(c3dev-test-dev) '\''
\export CONDA_SHLVL='\''2'\''
\export PATH='\''/home/tl/miniconda3/envs/c3dev-test-dev/bin:/home/tl/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'\'''
+ eval 'PS1='\''(c3dev-test-dev) '\''
\export CONDA_DEFAULT_ENV='\''c3dev-test-dev'\''
\export CONDA_PREFIX='\''/home/tl/miniconda3/envs/c3dev-test-dev'\''
\export CONDA_PREFIX_1='\''/home/tl/miniconda3'\''
\export CONDA_PROMPT_MODIFIER='\''(c3dev-test-dev) '\''
\export CONDA_SHLVL='\''2'\''
\export PATH='\''/home/tl/miniconda3/envs/c3dev-test-dev/bin:/home/tl/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'\'''
++ PS1='(c3dev-test-dev) '
++ export CONDA_DEFAULT_ENV=c3dev-test-dev
++ CONDA_DEFAULT_ENV=c3dev-test-dev
++ export CONDA_PREFIX=/home/tl/miniconda3/envs/c3dev-test-dev
++ CONDA_PREFIX=/home/tl/miniconda3/envs/c3dev-test-dev
++ export CONDA_PREFIX_1=/home/tl/miniconda3
++ CONDA_PREFIX_1=/home/tl/miniconda3
++ export 'CONDA_PROMPT_MODIFIER=(c3dev-test-dev) '
++ CONDA_PROMPT_MODIFIER='(c3dev-test-dev) '
++ export CONDA_SHLVL=2
++ CONDA_SHLVL=2
++ export PATH=/home/tl/miniconda3/envs/c3dev-test-dev/bin:/home/tl/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
++ PATH=/home/tl/miniconda3/envs/c3dev-test-dev/bin:/home/tl/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
+ __conda_hashr
+ '[' -n '' ']'
+ '[' -n '' ']'
+ hash -r
+ pip install -r /home/tl/pycogent3/condaenv.57qe8phq.requirements.txt
ERROR: Error installing 'file:///home/tl/pycogent3/PyCogent3Apps (from -r /home/tl/pycogent3/condaenv.57qe8phq.requirements.txt (line 12))': editable mode is not supported for pyproject.toml-style projects. pip is processing this project as pyproject.toml-style because it has a pyproject.toml file. Since the project has a setup.py and the pyproject.toml has no "build-backend" key for the "build_system" value, you may pass --no-use-pep517 to opt out of pyproject.toml-style processing. See PEP 517 for details on pyproject.toml-style projects.

[GavinHuttley]

Original comment by GavinH (Bitbucket: 557058:e40c23e1-e273-4527-a2f8-5de5876e870d, ).

---

That’s the one!

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

Great! I’ll tackle it when I get a chance tomorrow.

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

From all the articles/git forums I’ve been reading, this issue isn’t an uncommon one. I just stumbled on a potential fix, so I’ll give it a try and see how it goes.

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

Can I double check that you are using pip 19.1? The solution won’t work for older versions as I just found out… :disappointed:

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

Actually, even worse: https://github.com/pypa/pip/pull/6447

The pull request that is supposed to fix this problem isn’t merged yet.

[GavinHuttley]

Original comment by T La (Bitbucket: 5c73c2a3cfbc206be909a286, ).

---

So we are actually playing with bleeding edge stuff, and boy, keeping up with Python is harder than keeping up with the Jones. There are development versions of pip which claim to fix the issue, but I haven’t had any luck, nor do I think you want us to force users to install development versions of pip. As it stands currently, this is a bug in pip that is not yet resolved. We could revoke the toml while we wait for the upstream issue to be fixed, but its your call. Where to now?

[GavinHuttley]

Original comment by GavinH (Bitbucket: 557058:e40c23e1-e273-4527-a2f8-5de5876e870d, ).

---

Thanks for the research. Since this only affects developers, we just enforce pip version 18 (as I have in the revised conda environment file) until there’s a proper fix. Putting this Issue on hold until there's a proper fix to pip.