macro128 / pdm-conda

A PDM plugin to resolve/install/uninstall project dependencies with Conda
33 stars 1 forks source link

randomly showing `conda env not detected` or `can only concatenate tuple (not "list") to tuple` #31

Closed goyalyashpal closed 3 months ago

goyalyashpal commented 3 months ago

hi! thanks a lot for this app.

i am having trouble getting started with pdm-conda: it is randomly showing errors like:

[ProjectError]: Conda environment not detected.

or

ERROR: Failed to load plugin conda=pdm_conda:main: can only concatenate tuple
(not "list") to tuple

and like i have no clue whatsoever on how to how to solve it.

for the conda env not detected, i tried conda env list, but the env shown for current project under the $env:LOCALAPPDATA location doesn't have a name, so i don't know how can that be activated using conda activate <name>.

it might also be the case that it never really worked under the hood - just that the things i inferred implying that the conda's working fine with pdm was just pdm's task lol

> pdm info
[ProjectError]: Conda environment not detected.

> conda env list
# conda environments:
#
                         <$env:LOCALAPPDATA>\pdm\pdm\venvs\<$PWD-name>-iIJXwE5j-3.12
                         <...>\scoop\apps\mambaforge\24.3.0-0
base                     <...>\scoop\apps\mambaforge\current

here's how i created the venv:

> pdm install --plugins
WARNING: Project requires a python version of <3.13,>=3.12, The virtualenv is
INFO: python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at
<.>\.venv
Plugins are installed successfully into .pdm-plugins.

> pdm venv create -w conda
Virtualenv <$env:LOCALAPPDATA>\pdm\pdm\venvs\<$PWD>-iIJXwE5j-3.12 is created
successfully

attachments

# pyproject.toml's exerpt
[project]
dependencies = [
    # "keras",
    "tensorflow>=2.16",
]
requires-python = "<3.13,>=3.12"

[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

[tool.pdm]
distribution = true
plugins = [
    "pdm-conda",
]

[tool.pdm.conda]
channels = ["conda-forge/noarch", "conda-forge", "anaconda"]
as-default-manager = true
runner = "conda"

environment

> scoop list mambaforge   # to provide conda

Name       Version  Source Updated             Info
----       -------  ------ -------             ----
mambaforge 24.3.0-0 extras 2024-04-30 17:04:29

> which -a python
/d/UserFiles/scoop/shims/python
/d/ProgramFiles/Python/Python312/python
/mingw64/bin/python

> pdm --version
PDM, version 2.15.1
> $PSVersionTable.PSVersion    # PowerShell
5.1.19041.4170

> systeminfo | grep -Ei "^os name|system type"
OS Name:                   Microsoft Windows 10 Home Single Language
System Type:               x64-based PC
macro128 commented 3 months ago

Hi! It seems to be a Windows related problem, as I don't have a Windows machine I can only fix it with your help, if you want to do it please follow these instructions:

  1. Change you pyproject.toml to match this:
    [tool.pdm]
    distribution = true
    plugins = [
        "git+https://github.com/macro128/pdm-conda.git",
    ]
  2. Run pdm install --plugins
  3. Run pdm venv create -vv --cn test -w conda and show me the output
goyalyashpal commented 3 months ago
[tool.pdm]
distribution = true
plugins = [
    # "pdm-conda",
    "git+https://github.com/macro128/pdm-conda.git",
]
> pdm install --plugins
See $LOCALAPPDATA\pdm\pdm\Logs\pdm-install-plugins-h1lvvdp2.log for detailed
debug log.
[UnpackError]: remote: Repository not found.
fatal: repository 'https://github.com/macro128/base/' not found
fatal: clone of 'https://github.com/macro128/base' into submodule path
'$LOCALAPPDATA/Temp/pdm-build-ey1polne/base' failed
Failed to clone 'base'. Retry scheduled
remote: Repository not found.
fatal: repository 'https://github.com/macro128/base/' not found
fatal: clone of 'https://github.com/macro128/base' into submodule path
'$LOCALAPPDATA/Temp/pdm-build-ey1polne/base' failed
Failed to clone 'base' a second time, aborting

WARNING: Add '-v' to see the detailed traceback
macro128 commented 3 months ago

Okay, I think your problem is related to how Windows machines use environment variables and I might have fixed it, please try with

[tool.pdm]
distribution = true
plugins = [
    "pdm-conda==0.17.5-dev1",
]
goyalyashpal commented 3 months ago
> pdm install --plugins
Plugins are installed successfully into .pdm-plugins.

> # pdm venv create -vv --cn test -w conda
> pdm venv create -vv -cn test -w conda
Details

```console STATUS: Creating virtualenv using conda... Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "$USERPROFILE\.local\bin\pdm.exe\__main__.py", line 7, in File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 358, in main return core.main(args or sys.argv[1:]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 276, in main raise cast(Exception, err).with_traceback(traceback) from None File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 271, in main self.handle(project, options) File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 207, in handle command.handle(project, options) File "D:\....\.pdm-plugins\Lib\site-packages\pdm_conda\cli\commands\venv\create.py", line 61, in handle super().handle(project, options) File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\cli\commands\venv\create.py", line 50, in handle path = venv_backend.create( ^^^^^^^^^^^^^^^^^^^^ File "D:\....\.pdm-plugins\Lib\site-packages\pdm_conda\cli\commands\venv\backends.py", line 33, in create return super().create(name, args, force, in_project, prompt, with_pip) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\cli\commands\venv\backends.py", line 121, in create python_version=self.ident, ^^^^^^^^^^ File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\cli\commands\venv\backends.py", line 172, in ident return super().ident ^^^^^^^^^^^^^ File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\cli\commands\venv\backends.py", line 63, in ident return self._resolved_interpreter.identifier ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\ProgramFiles\Python\Python312\Lib\functools.py", line 995, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\cli\commands\venv\backends.py", line 48, in _resolved_interpreter for py_version in self.project.iter_interpreters(self.python, search_venv=False, filter_func=match_func): File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\project\core.py", line 653, in iter_interpreters for interpreter in self.find_interpreters(python_spec, search_venv): File "D:\....\.pdm-plugins\Lib\site-packages\pdm_conda\project\core.py", line 257, in find_interpreters if not i.get_venv().is_conda: ^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'is_conda' ```

macro128 commented 3 months ago

Sorry about that, try with pdm-conda==0.17.5-dev2

goyalyashpal commented 3 months ago

i think the wezterm terminal has eaten some of the output for the pdm venv create ... command as i think i read there was some "success" or "done", but i might be wrong anyhow, here's what i got:

> pdm install --plugins
Plugins are installed successfully into .pdm-plugins.

> pdm venv create -vv -cn test -w conda
STATUS: Creating virtualenv using conda...
No channels specified for creating environment, using defaults if exist.
pdm.termui: cmd: conda create -y --strict-channel-priority --json --prefix $LOCALAPPDATA\pdm\pdm\venvs\test python==3.12.* -c defaults --override-channels
> pdm info
[CondaExecutionError]: Error locking dependencies
usage: conda-script.py [-h] [-v] [--no-plugins] [-V] COMMAND ...
conda-script.py: error: unrecognized arguments: -n base

> pdm lock
[CondaExecutionError]: Error locking dependencies
usage: conda-script.py [-h] [-v] [--no-plugins] [-V] COMMAND ...
conda-script.py: error: unrecognized arguments: -n base

WARNING: Add '-v' to see the detailed traceback

> conda env list
# conda environments:
#
                         $LOCALAPPDATA\pdm\pdm\venvs\test
                         D:\UserFiles\scoop\apps\mambaforge\24.3.0-0
base                     D:\UserFiles\scoop\apps\mambaforge\current
Verbose Details

```console > pdm info -v Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "$USERPROFILE\.local\bin\pdm.exe\__main__.py", line 7, in File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 358, in main return core.main(args or sys.argv[1:]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 276, in main raise cast(Exception, err).with_traceback(traceback) from None File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 271, in main self.handle(project, options) File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 207, in handle command.handle(project, options) File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\cli\commands\info.py", line 32, in handle interpreter = project.environment.interpreter ^^^^^^^^^^^^^^^^^^^ File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\project\core.py", line 299, in environment self._environment = self.get_environment() ^^^^^^^^^^^^^^^^^^^^^^ File "D:\...\.pdm-plugins\Lib\site-packages\pdm_conda\project\core.py", line 215, in get_environment if not get_venv_like_prefix(self.python.executable)[1]: ^^^^^^^^^^^ File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\project\core.py", line 155, in python python = self.resolve_interpreter() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\project\core.py", line 239, in resolve_interpreter for _, venv in iter_venvs(self): File "D:\...\.pdm-plugins\Lib\site-packages\pdm_conda\cli\commands\venv\utils.py", line 50, in wrapper for python in find_pythons(project): File "D:\...\.pdm-plugins\Lib\site-packages\pdm_conda\cli\commands\venv\utils.py", line 22, in find_pythons base_env = conda_info(project, env_name="base")["base_env"] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\...\.pdm-plugins\Lib\site-packages\pdm_conda\conda.py", line 544, in conda_info info = run_conda(cmd) ^^^^^^^^^^^^^^ File "D:\...\.pdm-plugins\Lib\site-packages\pdm_conda\conda.py", line 152, in run_conda raise exception_cls(msg, **kwargs) from e pdm_conda.conda.CondaExecutionError: Error locking dependencies usage: conda-script.py [-h] [-v] [--no-plugins] [-V] COMMAND ... conda-script.py: error: unrecognized arguments: -n base ```


self note:

see, its cases like this (usage from script/wrappers) in cli applications where on unknown arguments, showing the original invokation command along with usage error might have helped.

but ofc, it would require removing any secrets from there too which would likely increase the hassle of creating such a output message.

goyalyashpal commented 3 months ago

just for info, in the conda env list, the base D:\UserFiles\scoop\apps\mambaforge\current is a symlink/junction to the D:\UserFiles\scoop\apps\mambaforge\24.3.0-0. i don't know if it matters or not.

that's due to how scoop package installer manages stuff.

macro128 commented 3 months ago

The good news is pdm venv create worked correctly so we're getting closer :), I think I patched all the paths correctly and fixed an error with Conda runner parser, please try dm-conda==0.17.5-dev4.


About the self note: actually it outputs the commands used if you add -vv flag

goyalyashpal commented 3 months ago

now it's back to Not a conda environment, though the pdm lock works

wezterm again ate somethings now... this time the command i entered.. so, i am just guessing that it was pdm list in 3rd command below. it also ate some lines when running my own python script too... 😕

Details

```console > pdm install --plugins Plugins are installed successfully into .pdm-plugins. > pdm venv create -vv -cn test -w conda STATUS: Creating virtualenv using conda... [VirtualenvCreateError]: The location $LOCALAPPDATA\pdm\pdm\venvs\test is not empty, add --force to overwrite it. > pdm list WARNING: Project requires a python version of <3.13,>=3.12, The virtualenv is being created for you as it cannot be matched to the right version. INFO: python.use_venv is on, creating a virtualenv for this project... Virtualenv is created successfully at $LOCALAPPDATA\pdm\pdm\venvs\...proj-3.12 [CondaExecutionError]: Error locking dependencies EnvironmentLocationNotFound: Not a conda environment: $LOCALAPPDATA\pdm\pdm\venvs\...proj-3.12\python.exe WARNING: Add '-v' to see the detailed traceback > pdm venv create -vv -cn test -w conda --force STATUS: Creating virtualenv using conda... pdm.termui: cmd: conda env remove -y --json --prefix $LOCALAPPDATA\pdm\pdm\venvs\test pdm.termui: cmd: conda create -y --strict-channel-priority --json --prefix $LOCALAPPDATA\pdm\pdm\venvs\test python==3.12.* -c defaults --override-channels Virtualenv $LOCALAPPDATA\pdm\pdm\venvs\test is created successfully ``` ```console > pdm info PDM version: 2.15.1 Python Interpreter: $LOCALAPPDATA\pdm\pdm\venvs\...proj-3.12\python.exe (3.12) Project Root: D:/... Local Packages: > pdm lock 🔒 Lock successful Changes are written to pdm.lock. ```

```console > pdm list [CondaExecutionError]: Error locking dependencies EnvironmentLocationNotFound: Not a conda environment: $LOCALAPPDATA\pdm\pdm\venvs\...proj-3.12\python.exe WARNING: Add '-v' to see the detailed traceback > pdm list -cn test pdm list: error: unrecognized arguments: -cn > pdm list --venv test [PdmUsageError]: No virtualenv with key 'test' is found, must be one of ['3.12']. You can create one with 'pdm venv create'. > pdm venv list Virtualenvs created with this project: - 3.12: $LOCALAPPDATA\pdm\pdm\venvs\...proj-3.12 > conda env list # conda environments: # $LOCALAPPDATA\pdm\pdm\venvs\...proj-3.12 $LOCALAPPDATA\pdm\pdm\venvs\test D:\UserFiles\scoop\apps\mambaforge\24.3.0-0 base D:\UserFiles\scoop\apps\mambaforge\current ```

Details

```console > pdm list -vv Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "$USERPROFILE\.local\bin\pdm.exe\__main__.py", line 7, in File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 358, in main return core.main(args or sys.argv[1:]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 276, in main raise cast(Exception, err).with_traceback(traceback) from None File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 271, in main self.handle(project, options) File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 207, in handle command.handle(project, options) File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\cli\commands\list.py", line 168, in handle packages = project.environment.get_working_set() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\...\.pdm-plugins\Lib\site-packages\pdm_conda\environments\conda.py", line 77, in get_working_set dist_map = working_set._dist_map | conda_list(self.project) ^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\...\.pdm-plugins\Lib\site-packages\pdm_conda\conda.py", line 581, in conda_list packages = run_conda(config.command("list") + ["--json"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\...\.pdm-plugins\Lib\site-packages\pdm_conda\conda.py", line 152, in run_conda raise exception_cls(msg, **kwargs) from e pdm_conda.conda.CondaExecutionError: Error locking dependencies EnvironmentLocationNotFound: Not a conda environment: $LOCALAPPDATA\pdm\pdm\venvs\...project-3.12\python.exe ```

macro128 commented 3 months ago

Hi! You should use pdm use --venv test after creating the venv to make pdm run the commands with that python

goyalyashpal commented 3 months ago

You should use pdm use --venv test

hi! like i said, pdm doesn't detect/remember the test venv as reflected in output of pdm venv list above. the result of pdm use aligns with this as shown below.

just for quick reference, i am re-sharing the output of pdm venv list and conda env list too

> pdm use --venv test
[PdmUsageError]: No virtualenv with key 'test' is found, must be one of
['3.12'].
You can create one with 'pdm venv create'.

> pdm venv list
Virtualenvs created with this project:

-  3.12: $LOCALAPPDATA\pdm\pdm\venvs\...proj-3.12

> conda env list
# conda environments:
#
                         $LOCALAPPDATA\pdm\pdm\venvs\...proj-3.12
                         $LOCALAPPDATA\pdm\pdm\venvs\test
                         D:\UserFiles\scoop\apps\mambaforge\24.3.0-0
base                     D:\UserFiles\scoop\apps\mambaforge\current
macro128 commented 3 months ago

Do the path $LOCALAPPDATA\pdm\pdm\venvs\test exists with $LOCALAPPDATA? If it does, there's something weird with environment variables

goyalyashpal commented 3 months ago

Do the path $LOCALAPPDATA\pdm\pdm\venvs\test exists with $LOCALAPPDATA?

hi!

the $ notation is just bleed off of my habbit of using msys2's bash.

macro128 commented 3 months ago

Hi! I was just modifying my previous message, I got a Windows machine and fixed all the problems I could find, I tried using scoop and conda/mamba and it worked correctly, if you want to give it a try, use pdm-conda==0.17.5-dev6

goyalyashpal commented 3 months ago

this time it errored out at (what seems to be) the very last step i.e. pdm list, and ran successfully upto pdm use --venv test

Preceeding succesful commands

```console $ pdm install --plugins Plugins are installed successfully into .pdm-plugins. $ pdm venv create -vv -cn test -w conda --force STATUS: Creating virtualenv using conda... pdm.termui: cmd: conda info --base --json pdm.termui: cmd: conda env remove -y --json --prefix $LOCALAPPDATA\pdm\pdm\venvs\test pdm.termui: cmd: conda create -y --strict-channel-priority --json --prefix $LOCALAPPDATA\pdm\pdm\venvs\test python==3.12.* -c defaults --override-channels Virtualenv $LOCALAPPDATA\pdm\pdm\venvs\test is created successfully $ pdm venv list Virtualenvs created with this project: - 3.12: $LOCALAPPDATA\pdm\pdm\venvs\...proj-3.12 - ...proj-3.12: $LOCALAPPDATA\pdm\pdm\venvs\...proj-3.12 - test: $LOCALAPPDATA\pdm\pdm\venvs\test $ pdm use --venv test Using Python interpreter: $LOCALAPPDATA\pdm\pdm\venvs\test\python.exe (3.12) $ pdm lock 🔒 Lock successful Changes are written to pdm.lock. ```

$ pdm list
[CondaExecutionError]: Error locking dependencies
EnvironmentLocationNotFound: Not a conda environment: $LOCALAPPDATA\pdm\pdm\venvs\test\python.exe
WARNING: Add '-v' to see the detailed traceback

$ pdm list -vv
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "$USERPROFILE\.local\bin\pdm.exe\__main__.py", line 7, in <module>
  File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 358, in main
    return core.main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 276, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 271, in main
    self.handle(project, options)
  File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\core.py", line 207, in handle
    command.handle(project, options)
  File "D:\UserFiles\pipx\venvs\pdm\Lib\site-packages\pdm\cli\commands\list.py", line 168, in handle
    packages = project.environment.get_working_set()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\College\6-Sem\PROJ-CS-302_Project_1\.pdm-plugins\Lib\site-packages\pdm_conda\environments\conda.py", line 85, in get_working_set
    dist_map = working_set._dist_map | conda_list(self.project)
                                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\College\6-Sem\PROJ-CS-302_Project_1\.pdm-plugins\Lib\site-packages\pdm_conda\conda.py", line 581, in conda_list
    packages = run_conda(config.command("list") + ["--json"])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\College\6-Sem\PROJ-CS-302_Project_1\.pdm-plugins\Lib\site-packages\pdm_conda\conda.py", line 152, in run_conda
    raise exception_cls(msg, **kwargs) from e
pdm_conda.conda.CondaExecutionError: Error locking dependencies
EnvironmentLocationNotFound: Not a conda environment: $LOCALAPPDATA\pdm\pdm\venvs\test\python.exe
goyalyashpal commented 3 months ago

successful yayyy! thanks a lot. 🪔🪔🪔 ur patience is wonderful.

> # This took 3 minutes
> pdm venv create -vv -cn test -w conda --force
STATUS: Creating virtualenv using conda...
pdm.termui: cmd: conda info --base --json
pdm.termui: cmd: conda create -y --strict-channel-priority --json --prefix $LOCALAPPDATA\pdm\pdm\venvs\test python==3.12.* -c defaults --override-channels
Virtualenv $LOCALAPPDATA\pdm\pdm\venvs\test is created successfully

> # Works :)
> pdm venv list
Virtualenvs created with this project:

-  test: $LOCALAPPDATA\pdm\pdm\venvs\test

> # This worked by default; `pdm use --venv test` was not required 😀
> pdm list
INFO: Virtualenv $LOCALAPPDATA\pdm\pdm\venvs\test is reused.
╭─────────────────┬─────────────┬──────────╮
│ name            │ version     │ location │
├─────────────────┼─────────────┼──────────┤
│ bzip2           │ 1.0.8       │          │
│ ca-certificates │ 2024.3.11   │          │
│ expat           │ 2.6.2       │          │
│ libffi          │ 3.4.4       │          │
  ...

> pdm lock
🔒 Lock successful
Changes are written to pdm.lock.

> # Works :)
> pdm sync
Synchronizing working set with resolved packages: 36 to add, 0 to update, 0 to 
remove

  ✔ Install certifi 2024.2.2 successful
  ✔ Install idna 3.7 successful
  ✔ Install astunparse 1.6.3 successful
  ...
Installing the project as an editable package...
  ✔ Install <ProjName> 0.1.0 successful

🎉 All complete!