An error occurred while retrieving available packages

johnne commented 1 year ago

Description

I'm getting an error message saying "An error occurred while retrieving available packages" as well as "I know you want to give up, but wait a bit longer.." which never ends. In my terminal I get

[D 2023-05-05 07:59:37.313 ServerApp.mamba_gator] ...
[D 2023-05-05 07:59:39.573 ServerApp.mamba_gator] Accepting token-authenticated connection from 127.0.0.1
[D 2023-05-05 07:59:39.592 ServerApp.mamba_gator] 202 GET /conda/tasks/3?1683266378066 (127.0.0.1) 19.02ms
[E 2023-05-05 07:59:39.626 ServerApp.mamba_gator] Error for task {'type': 'InvalidVersion', 'error': "Invalid version: '2022.02.02.gr3.9'", 'message': 'InvalidVersion("Invalid version: \'2022.02.02.gr3.9\'")', 'traceback': ['  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 96, in execute_task\n    result = await f(*args)\n             ^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 380, in update_available\n    answer = await env_manager.list_available()\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 692, in list_available\n    packages = await current_loop.run_in_executor(None, format_packages, data)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/concurrent/futures/thread.py", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 662, in format_packages\n    version = parse(entry.get("version", ""))\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 52, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 198, in __init__\n    raise InvalidVersion(f"Invalid version: \'{version}\'")\n']}.
[D 2023-05-05 07:59:40.603 ServerApp.mamba_gator] Accepting token-authenticated connection from 127.0.0.1
[D 2023-05-05 07:59:40.604 ServerApp.mamba_gator] {'type': 'InvalidVersion', 'error': "Invalid version: '2022.02.02.gr3.9'", 'message': 'InvalidVersion("Invalid version: \'2022.02.02.gr3.9\'")', 'traceback': ['  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 96, in execute_task\n    result = await f(*args)\n             ^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 380, in update_available\n    answer = await env_manager.list_available()\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 692, in list_available\n    packages = await current_loop.run_in_executor(None, format_packages, data)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/concurrent/futures/thread.py", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 662, in format_packages\n    version = parse(entry.get("version", ""))\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 52, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 198, in __init__\n    raise InvalidVersion(f"Invalid version: \'{version}\'")\n']}
[E 2023-05-05 07:59:40.605 ServerApp.mamba_gator] {
      "Host": "localhost:8890",
      "Accept": "*/*",
      "Referer": "http://localhost:8890/lab",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/112.0"
    }

I'm able to run both conda search --json and mamba search --json which I saw could be a problem in another issue (#127)

Reproduce

Activate Conda environment with jupyterlab and gator_lab installed
Run jupyter lab

Expected behavior

No error messages

Context

Python package version: Version: 3.11.3 Build: h1456518_0_cpython

Extension version: Version: 5.2.0 Build: pyhd8ed1ab_0


jupyter labextension list
Config option `kernel_spec_manager_class` not recognized by `ListLabExtensionsApp`.
[W 2023-05-05 08:02:54.466 LabApp] Config option `kernel_spec_manager_class` not recognized by `LabApp`.
JupyterLab v3.6.3
/Users/johnlarsson/mambaforge/envs/jupyter/share/jupyter/labextensions
    jupyterlab_pygments v0.2.2 enabled OK (python, jupyterlab_pygments)
    nbdime-jupyterlab v2.2.0 enabled OK
    jupyterlab_templates v0.4.0 enabled OK (python, jupyterlab_templates)
    @mamba-org/gator-lab v3.1.0 enabled OK (python, mamba_gator)
    @voila-dashboards/jupyterlab-preview v2.2.0 enabled OK (python, voila)
    @jupyterlab/github v3.0.1 enabled OK (python, jupyterlab_github)
    @jupyterlab/git v0.41.0 enabled OK (python, jupyterlab-git)
    @jupyter-widgets/jupyterlab-manager v5.0.7 enabled OK (python, jupyterlab_widgets)

Other labextensions (built into JupyterLab) app dir: /Users/johnlarsson/mambaforge/envs/jupyter/share/jupyter/lab @quarto/jupyterlab-quarto v0.1.42 enabled OK

- General information:
```bash
conda info 

     active environment : jupyter
    active env location : /Users/johnlarsson/mambaforge/envs/jupyter
            shell level : 2
       user config file : /Users/johnlarsson/.condarc
 populated config files : /Users/johnlarsson/mambaforge/.condarc
                          /Users/johnlarsson/.condarc
          conda version : 23.3.1
    conda-build version : not installed
         python version : 3.10.9.final.0
       virtual packages : __archspec=1=arm64
                          __osx=13.3.1=0
                          __unix=0=0
       base environment : /Users/johnlarsson/mambaforge  (writable)
      conda av data dir : /Users/johnlarsson/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/bioconda/osx-arm64
                          https://conda.anaconda.org/bioconda/noarch
          package cache : /Users/johnlarsson/mambaforge/pkgs
                          /Users/johnlarsson/.conda/pkgs
       envs directories : /Users/johnlarsson/mambaforge/envs
                          /Users/johnlarsson/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.3.1 requests/2.28.2 CPython/3.10.9 Darwin/22.4.0 OSX/13.3.1
                UID:GID : 502:20
             netrc file : None
           offline mode : False

Command Line Output

[D 2023-05-05 07:59:37.313 ServerApp.mamba_gator] ...
[D 2023-05-05 07:59:39.573 ServerApp.mamba_gator] Accepting token-authenticated connection from 127.0.0.1
[D 2023-05-05 07:59:39.592 ServerApp.mamba_gator] 202 GET /conda/tasks/3?1683266378066 (127.0.0.1) 19.02ms
[E 2023-05-05 07:59:39.626 ServerApp.mamba_gator] Error for task {'type': 'InvalidVersion', 'error': "Invalid version: '2022.02.02.gr3.9'", 'message': 'InvalidVersion("Invalid version: \'2022.02.02.gr3.9\'")', 'traceback': ['  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 96, in execute_task\n    result = await f(*args)\n             ^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 380, in update_available\n    answer = await env_manager.list_available()\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 692, in list_available\n    packages = await current_loop.run_in_executor(None, format_packages, data)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/concurrent/futures/thread.py", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 662, in format_packages\n    version = parse(entry.get("version", ""))\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 52, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 198, in __init__\n    raise InvalidVersion(f"Invalid version: \'{version}\'")\n']}.
[D 2023-05-05 07:59:40.603 ServerApp.mamba_gator] Accepting token-authenticated connection from 127.0.0.1
[D 2023-05-05 07:59:40.604 ServerApp.mamba_gator] {'type': 'InvalidVersion', 'error': "Invalid version: '2022.02.02.gr3.9'", 'message': 'InvalidVersion("Invalid version: \'2022.02.02.gr3.9\'")', 'traceback': ['  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 96, in execute_task\n    result = await f(*args)\n             ^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/handlers.py", line 380, in update_available\n    answer = await env_manager.list_available()\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 692, in list_available\n    packages = await current_loop.run_in_executor(None, format_packages, data)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/concurrent/futures/thread.py", line 58, in run\n    result = self.fn(*self.args, **self.kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/mamba_gator/envmanager.py", line 662, in format_packages\n    version = parse(entry.get("version", ""))\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 52, in parse\n    return Version(version)\n           ^^^^^^^^^^^^^^^^\n', '  File "/Users/johnlarsson/mambaforge/envs/jupyter/lib/python3.11/site-packages/packaging/version.py", line 198, in __init__\n    raise InvalidVersion(f"Invalid version: \'{version}\'")\n']}
[E 2023-05-05 07:59:40.605 ServerApp.mamba_gator] {
      "Host": "localhost:8890",
      "Accept": "*/*",
      "Referer": "http://localhost:8890/lab",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/112.0"
    }

Browser Output

Paste the output from your browser Javascript console here.

fcollonval commented 1 year ago

Thanks for reporting the issue @johnne

The error originated from

https://github.com/mamba-org/gator/blob/02795700431f2e640178f752311c3035f1654cd5/mamba_gator/envmanager.py#L662

The easy fix would be to ignore invalid version number. Then a follow up could be to handle invalid version.

Would you be willing to open a PR? The first step would be to wrap the version parsing with a try-except:

try:
    version = parse(entry.get("version", "")) 
except InvalidVersion:
    name = entry.get("name")
    version = entry.get("version")
    self.log.warning(f"Unable to parse version '{version}' of '{name}'")
    continue

With InvalidVersion to be imported from packaging.version.

johnne commented 1 year ago

Thank you! I edited the installation I have and that seems to get rid of the error message.

I will open a PR

fcollonval commented 1 year ago

Thanks for trying the fix. Would you mind posting here the log about which package failed to be parsed?

johnne commented 1 year ago

unparsed_versions.txt

This is what I get when running jupyter lab --debug and doing grep "Unable to parse version" from the debug log.

fcollonval commented 1 year ago

Thanks a lot @johnne

So as expected most packages failing are none-python ones as they do not require to conform to Python version syntax. And in recent version of packaging those are not allowed. The fix you made (thanks again) will avoid the bug but prevent to handle those packages. Would you be willing to do a follow-up PR to handle version for those packages?

Basically we should internalize the old LegacyVersion from packaging that was the fallback when failing to parse a version string.

See: https://github.com/pypa/packaging/blob/20.9/packaging/version.py

The code was

    try:
        return Version(version)
    except InvalidVersion:
        return LegacyVersion(version)

So the follow-up PR would be to change the try-except introduced in #202 by:

raw_version = entry.get("version", "")
try:
    version = parse(raw_version) 
except InvalidVersion:
    name = entry.get("name")
    msg = f"Falling back to 'LegacyVersion' for version '{raw_version}' of '{name}'"
    self.log.debug(msg)
    version = LegacyVersion(raw_version)

and copying the definition of LegacyVersion (and all the code used by it) from https://github.com/pypa/packaging/blob/20.9/packaging/version.py

Would you be willing to do the follow-up PR?

FYI I released 5.2.1 with your fix.

mamba-org / gator