Vendored poetry monkeypatches subprocess.run and Popen

AlbertDeFusco commented 1 year ago

Checklist

[X] I added a descriptive title
[X] I searched open reports and couldn't find a duplicate

What happened?

I know that this is a strange bug report and that the problem is not conda-lock's fault. I wonder if a new version of poetry will help, even slightly. This compatibility code was removed for version 1.2.

I've noticed that when conda_lock is imported then subprocess.run no longer supports capture_ouptut=True. The error messages trace to the vendored version of poetry included with conda-lock.

In [1]: import subprocess

In [2]: subprocess.run(['/usr/bin/env', 'echo', '$SHELL'], capture_output=True).stdout
Out[2]: b'$SHELL\n'

In [3]: import conda_lock

In [4]: subprocess.run(['/usr/bin/env', 'echo', '$SHELL'], capture_output=True).stdout
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[4], line 1
----> 1 subprocess.run(['/usr/bin/env', 'echo', '$SHELL'], capture_output=True).stdout

File ~/cl/lib/python3.10/site-packages/conda_lock/_vendor/poetry/utils/_compat.py:200, in run(*popenargs, **kwargs)
    197         raise ValueError("stdin and input arguments may not both be used.")
    198     kwargs["stdin"] = PIPE
--> 200 process = Popen(*popenargs, **kwargs)
    201 try:
    202     process.__enter__()  # No-Op really... illustrate "with in 2.4"

TypeError: Popen.__init__() got an unexpected keyword argument 'capture_output'

Conda Info

active environment : /Users/adefusco/cl
    active env location : /Users/adefusco/cl
            shell level : 2
       user config file : /Users/adefusco/.condarc
 populated config files : /Users/adefusco/.condarc
          conda version : 23.3.1
    conda-build version : 3.23.3
         python version : 3.9.13.final.0
       virtual packages : __archspec=1=arm64
                          __osx=13.2.1=0
                          __unix=0=0
       base environment : /Users/adefusco/Applications/miniconda3  (writable)
      conda av data dir : /Users/adefusco/Applications/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/adefusco/Applications/miniconda3/pkgs
                          /Users/adefusco/.conda/pkgs
       envs directories : /Users/adefusco/Applications/miniconda3/envs
                          /Users/adefusco/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.3.1 requests/2.28.1 CPython/3.9.13 Darwin/22.3.0 OSX/13.2.1
                UID:GID : 502:20
             netrc file : None
           offline mode : False

Conda Config

==> /Users/adefusco/.condarc <==
extra_safety_checks: False
ssl_verify: True

Conda list

# packages in environment at /Users/adefusco/cl:
#
# Name                    Version                   Build  Channel
appdirs                   1.4.4              pyhd3eb1b0_0    defaults
appnope                   0.1.2           py310hca03da5_1001    defaults
asttokens                 2.0.5              pyhd3eb1b0_0    defaults
backcall                  0.2.0              pyhd3eb1b0_0    defaults
brotlipy                  0.7.0           py310h1a28f6b_1002    defaults
bzip2                     1.0.8                h620ffc9_4    defaults
ca-certificates           2023.01.10           hca03da5_0    defaults
cachecontrol              0.12.11         py310hca03da5_1    defaults
cachecontrol-with-filecache 0.12.11         py310hca03da5_1    defaults
cachy                     0.3.0              pyhd3eb1b0_0    defaults
certifi                   2022.12.7       py310hca03da5_0    defaults
cffi                      1.15.1          py310h80987f9_3    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
click                     8.0.4           py310hca03da5_0    defaults
click-default-group       1.2.2           py310hca03da5_0    defaults
clikit                    0.6.2                      py_0    defaults
conda-lock                1.4.0           py310hca03da5_0    defaults
crashtest                 0.3.1              pyhd3eb1b0_1    defaults
cryptography              39.0.1          py310h834c97f_0    defaults
decorator                 5.1.1              pyhd3eb1b0_0    defaults
distlib                   0.3.6           py310hca03da5_0    defaults
ensureconda               1.4.3           py310hca03da5_0    defaults
executing                 0.8.3              pyhd3eb1b0_0    defaults
filelock                  3.9.0           py310hca03da5_0    defaults
html5lib                  1.1                pyhd3eb1b0_0    defaults
idna                      3.4             py310hca03da5_0    defaults
importlib-metadata        6.0.0           py310hca03da5_0    defaults
importlib_metadata        6.0.0                hd3eb1b0_0    defaults
ipython                   8.12.0          py310hca03da5_0    defaults
jaraco.classes            3.2.1              pyhd3eb1b0_0    defaults
jedi                      0.18.1          py310hca03da5_1    defaults
jinja2                    3.1.2           py310hca03da5_0    defaults
keyring                   23.13.1         py310hca03da5_0    defaults
libcxx                    14.0.6               h848a8c0_0    defaults
libffi                    3.4.2                hca03da5_6    defaults
lockfile                  0.12.2          py310hca03da5_0    defaults
markupsafe                2.1.1           py310h1a28f6b_0    defaults
matplotlib-inline         0.1.6           py310hca03da5_0    defaults
more-itertools            8.12.0             pyhd3eb1b0_0    defaults
msgpack-python            1.0.3           py310h525c30c_0    defaults
ncurses                   6.4                  h313beb8_0    defaults
openssl                   1.1.1t               h1a28f6b_0    defaults
packaging                 23.0            py310hca03da5_0    defaults
parso                     0.8.3              pyhd3eb1b0_0    defaults
pastel                    0.2.1                      py_0    defaults
pexpect                   4.8.0              pyhd3eb1b0_3    defaults
pickleshare               0.7.5           pyhd3eb1b0_1003    defaults
pip                       23.0.1          py310hca03da5_0    defaults
pkginfo                   1.9.6           py310hca03da5_0    defaults
platformdirs              2.5.2           py310hca03da5_0    defaults
prompt-toolkit            3.0.36          py310hca03da5_0    defaults
ptyprocess                0.7.0              pyhd3eb1b0_2    defaults
pure_eval                 0.2.2              pyhd3eb1b0_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pydantic                  1.10.2          py310h1a28f6b_0    defaults
pygments                  2.11.2             pyhd3eb1b0_0    defaults
pylev                     1.3.0                      py_0    defaults
pyopenssl                 23.0.0          py310hca03da5_0    defaults
pysocks                   1.7.1           py310hca03da5_0    defaults
python                    3.10.10              hc0d8a6c_2    defaults
pyyaml                    6.0             py310h80987f9_1    defaults
readline                  8.2                  h1a28f6b_0    defaults
requests                  2.28.1          py310hca03da5_1    defaults
ruamel.yaml               0.17.21         py310h1a28f6b_0    defaults
ruamel.yaml.clib          0.2.6           py310h1a28f6b_1    defaults
setuptools                66.0.0          py310hca03da5_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
sqlite                    3.41.2               h80987f9_0    defaults
stack_data                0.2.0              pyhd3eb1b0_0    defaults
tk                        8.6.12               hb8d0fd4_0    defaults
tomli                     2.0.1           py310hca03da5_0    defaults
tomlkit                   0.11.1          py310hca03da5_0    defaults
toolz                     0.12.0          py310hca03da5_0    defaults
traitlets                 5.7.1           py310hca03da5_0    defaults
typing-extensions         4.5.0           py310hca03da5_0    defaults
typing_extensions         4.5.0           py310hca03da5_0    defaults
tzdata                    2023c                h04d1e81_0    defaults
urllib3                   1.26.15         py310hca03da5_0    defaults
virtualenv                20.17.1         py310hca03da5_0    defaults
wcwidth                   0.2.5              pyhd3eb1b0_0    defaults
webencodings              0.5.1           py310hca03da5_1    defaults
wheel                     0.38.4          py310hca03da5_0    defaults
xz                        5.2.10               h80987f9_1    defaults
yaml                      0.2.5                h1a28f6b_0    defaults
zipp                      3.11.0          py310hca03da5_0    defaults
zlib                      1.2.13               h5a0b063_0    defaults

Additional Context

I am a contributor to conda-project where conda-lock is utilized as a library. I noticed this error occurs when an environment containing conda-project also contains setupools-scm. Then when conda-project imports conda-lock something happens that setuptools-scm wants to do subprocess.run(capture_output=True) and this happens after the conda-lock-vendored poetry is imported.

maresb commented 1 year ago

Hi @AlbertDeFusco, thanks so much for the report!

I was confused at first since the monkeypatch you describe is within the if PY34: block. But the definition of PY34 is sys.version_info >= (3, 4), so that block is indeed running.

I agree that revendoring Poetry would help. I hope to get around to this soon.

riccardoporreca commented 1 year ago

@maresb, @AlbertDeFusco, I am experiencing the same problem, which is biting quite hard

The issue is indeed related to the vendored Poetry, which has dropped most of utils/_compat.py in https://github.com/python-poetry/poetry/pull/3405/files#diff-58970a6af112ab5dce31955e7d73936f8ad104ef7ae7d1ecf677f729e60206f2

Below a reproducible example of the behavior to help clarifying when this bites.

In a nutshell, on Python 3.9 and 3.10 the presence of setuptool-scm and corresponding configuration (even empty) in pyproject.toml causes the issue, triggered by the vendored poetry. And this is completely unrelated to running conda-lock and making explicit usage of pyproject.toml as a source of dependencies.

Any plan to update the vendored poetry? Or any quick workaround/fix to address the problem?

We are planning to transition a large set of projects to conda-lock 2x from conda-lock1.0.5 and this is a major blocker. If there is anything I can help with, I will be happy to do so.

Python 3.9

mamba create -n conda-lock-test python=3.9 conda-lock setuptools-scm -c conda-forge --override-channels
conda activate conda-lock-test

mkdir conda-lock-test
cd conda-lock-test
git init
# all good so far
conda-lock --version
### conda-lock, version 2.1.0
echo "[tool.setuptools_scm]" > pyproject.toml
### TypeError: __init__() got an unexpected keyword argument 'capture_output'
# the actuall occurs at import time
python -c "import conda_lock"
### TypeError: Popen.__init__() got an unexpected keyword argument 'capture_output'

Note that an empty pyrpoject.toml or a pyproject.toml w/o [tool.setuptools_scm] would not harm

Python 3.10

Same behavior

Python 3.11

No issues

riccardoporreca commented 1 year ago

If this helps: Poetry 1.2 should be enough for this, since https://github.com/python-poetry/poetry/pull/3405 was included starting from 1.2.0a1

maresb commented 1 year ago

Yes, sorry about this. I have been meaning to do this for quite some time, and it's very high on my todo list. I'm hoping to get to it this weekend, but I can't promise anything.

riccardoporreca commented 1 year ago

Thanks @maresb, much appreciated, and if there is anything I can help with, especially around testing, let me know!

riccardoporreca commented 1 year ago

@maresb, would you have a realistic timeline for revendoring poetry and releasing a new conda-lock version?

Not trying to put pressure on you, but rather to manage internally expectations around a timeline for the adoption of conda-lock 2.x.

maresb commented 1 year ago

I'll try again this weekend, but since something might come up, better to assume that I'll get it by the subsequent weekend with about p=75%.

robertodr commented 11 months ago

I'm also getting hit by this, exactly as @riccardoporreca described.

MarosMK commented 9 months ago

Hi @maresb, this still seems to be an issue. Are we planning to update the vendored Poetry anytime soon?

maresb commented 9 months ago

Thanks for the reminder. This is still on my radar, and I would like very much to do the update soon, but it requires a big block of time that's very difficult for me to allocate at the moment. :disappointed:

riccardoporreca commented 9 months ago

@maresb, let me know if there is anything I can help with: I understand this is quite delicate and time consuming, but if there is even some preparatory work we can help with, happy to support!

MarosMK commented 6 months ago

Hi @maresb, just a reminder so it doesn't get forgotten.

AlbertDeFusco commented 3 months ago

Just so I don't forget I found another edge case with the vendored poetry. Is there a safe way to manage the import of conda_lock so that these cases might be avoided?

❯ conda list "conda-lock|attrs"
# packages in environment at /Users/adefusco/Desktop/cl:
#
# Name                    Version                   Build  Channel
attrs                     23.1.0           py39hca03da5_0  
conda-lock                2.5.6            py39hca03da5_0

This version of attrs has AttrsInstance

❯ python
Python 3.9.19 (main, May  6 2024, 14:39:30) 
[Clang 14.0.6 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from attrs import AttrsInstance
>>>

but the vendored version from conda-lock does not

❯ python
Python 3.9.19 (main, May  6 2024, 14:39:30) 
[Clang 14.0.6 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import conda_lock
>>> from attrs import AttrsInstance
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/adefusco/Desktop/cl/lib/python3.9/site-packages/attrs/__init__.py", line 3, in <module>
    from attr import (
ImportError: cannot import name 'AttrsInstance' from 'attr' (/Users/adefusco/Desktop/cl/lib/python3.9/site-packages/conda_lock/_vendor/poetry/core/_vendor/attr/__init__.py)
>>>

maresb commented 2 months ago

This will be closed by https://github.com/conda/conda-lock/pull/637 once we fix the remaining failing test case.

@AlbertDeFusco, that branch also fixes your import issue.

conda / conda-lock