Unable to cache available package list in jupyter when mamba is installed

tlvu commented 2 years ago

Description

Since we added mamba into our conda env, we are unable to update the cache of available package list. The extension used to work properly before. I removed mamba and the extension works again !

Screenshot from 2022-01-12 23-11-27

Screenshot from 2022-01-12 23-19-17

Screenshot from 2022-01-12 23-19-55

{
   "command" : "/opt/conda/bin/mamba repoquery search * --json",
   "conda_info" : {
      "GID" : 1000,
      "UID" : 1000,
      "active_prefix" : null,
      "active_prefix_name" : null,
      "av_data_dir" : "/opt/conda/etc/conda",
      "av_metadata_url_base" : null,
      "channels" : [
         "https://repo.anaconda.com/pkgs/main/linux-64",
         "https://repo.anaconda.com/pkgs/main/noarch",
         "https://repo.anaconda.com/pkgs/r/linux-64",
         "https://repo.anaconda.com/pkgs/r/noarch"
      ],
      "conda_build_version" : "not installed",
      "conda_env_version" : "4.11.0",
      "conda_location" : "/opt/conda/lib/python3.9/site-packages/conda",
      "conda_prefix" : "/opt/conda",
      "conda_private" : false,
      "conda_shlvl" : -1,
      "conda_version" : "4.11.0",
      "config_files" : [],
      "default_prefix" : "/opt/conda",
      "env_vars" : {
         "CIO_TEST" : "<not set>",
         "CONDA_ROOT" : "/opt/conda",
         "CURL_CA_BUNDLE" : "<not set>",
         "PATH" : "/opt/conda/envs/birdy/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
         "REQUESTS_CA_BUNDLE" : "<not set>",
         "SSL_CERT_FILE" : "<not set>"
      },
      "envs" : [],
      "envs_dirs" : [
         "/opt/conda/envs",
         "/home/jenkins/.conda/envs"
      ],
      "netrc_file" : null,
      "offline" : false,
      "pkgs_dirs" : [
         "/opt/conda/pkgs",
         "/home/jenkins/.conda/pkgs"
      ],
      "platform" : "linux-64",
      "python_version" : "3.9.5.final.0",
      "rc_path" : "/home/jenkins/.condarc",
      "requests_version" : "2.26.0",
      "root_prefix" : "/opt/conda",
      "root_writable" : true,
      "site_dirs" : [],
      "sys.executable" : "/opt/conda/bin/python",
      "sys.prefix" : "/opt/conda",
      "sys.version" : "3.9.5 (default, Jun  4 2021, 12:28:51) \n[GCC 7.5.0]",
      "sys_rc_path" : "/opt/conda/.condarc",
      "user_agent" : "conda/4.11.0 requests/2.26.0 CPython/3.9.5 Linux/4.15.0-163-generic debian/10 glibc/2.28",
      "user_rc_path" : "/home/jenkins/.condarc",
      "virtual_pkgs" : [
         [
            "__linux",
            "4.15.0",
            "0"
         ],
         [
            "__glibc",
            "2.28",
            "0"
         ],
         [
            "__unix",
            "0",
            "0"
         ],
         [
            "__archspec",
            "1",
            "x86_64"
         ]
      ]
   },
   "error" : "RuntimeError('LockFile error. Aborting.')",
   "exception_name" : "RuntimeError",
   "exception_type" : "<class 'RuntimeError'>",
   "traceback" : "Traceback (most recent call last):\n  File \"/opt/conda/lib/python3.9/site-packages/conda/exceptions.py\", line 1080, in __call__\n    return func(*args, **kwargs)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 917, in exception_converter\n    raise e\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 911, in exception_converter\n    exit_code = _wrapped_main(*args, **kwargs)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 869, in _wrapped_main\n    result = do_call(args, p)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 744, in do_call\n    exit_code = repoquery(args, parser)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 686, in repoquery\n    pool = repoquery_api.create_pool(channels, platform, use_installed)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/repoquery.py\", line 47, in create_pool\n    load_channels(\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/utils.py\", line 122, in load_channels\n    index = get_index(\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/utils.py\", line 103, in get_index\n    is_downloaded = dlist.download(True)\nRuntimeError: LockFile error. Aborting.\n"
}

Reproduce

Go to '...'
Click on '...'
Scroll down to '...'
See error '...'

Expected behavior

Context

Python package version: Python 3.7.12
Extension version: mamba_gator 5.1.2 pyhd8ed1ab_0 conda-forge

General information:


$ conda run -n birdy jupyter labextension list
JupyterLab v3.2.5
/opt/conda/envs/birdy/share/jupyter/labextensions
    jupyter-leaflet v0.15.0 enabled OK
    jupyterlab-plotly v5.5.0 enabled OK
    jupyterlab-topbar-extension v0.6.1 enabled OK (python, jupyterlab-topbar)
    jupyterlab-jupytext v1.3.8 enabled OK (python, jupytext)
    jupyterlab-logout v0.5.0 enabled OK (python, jupyterlab-logout)
    jupyterlab-system-monitor v0.8.0 enabled OK (python, jupyterlab-system-monitor)
    nbdime-jupyterlab v2.1.1 enabled OK
    @mamba-org/gator-lab v3.0.2 enabled OK (python, mamba_gator)
    @jupyterlab/git v0.34.0 enabled OK (python, jupyterlab-git)
    @bokeh/jupyter_bokeh v3.0.4 enabled OK (python, jupyter_bokeh)
    @voila-dashboards/jupyterlab-preview v2.1.0 enabled OK (python, voila)
    @jupyter-server/resource-usage v0.5.0 enabled OK (python, jupyter-resource-usage)
    @pyviz/jupyterlab_pyviz v2.1.0 enabled OK (python, pyviz_comms)
    @jupyter-widgets/jupyterlab-manager v3.0.1 enabled OK (python, jupyterlab_widgets)
    @hadim/jupyter-archive v3.2.1 enabled OK (python, jupyter-archive)

Other labextensions (built into JupyterLab) app dir: /opt/conda/envs/birdy/share/jupyter/lab jupyterlab-dash v0.4.0 enabled OK

$ conda info

 active environment : base
active env location : /opt/conda
        shell level : 1
   user config file : /root/.condarc

populated config files : /root/.condarc conda version : 4.11.0 conda-build version : not installed python version : 3.9.5.final.0 virtual packages : linux=4.15.0=0 glibc=2.28=0 unix=0=0 archspec=1=x86_64 base environment : /opt/conda (writable) conda av data dir : /opt/conda/etc/conda conda av metadata url : None channel URLs : https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /opt/conda/pkgs /root/.conda/pkgs envs directories : /opt/conda/envs /root/.conda/envs platform : linux-64 user-agent : conda/4.11.0 requests/2.26.0 CPython/3.9.5 Linux/4.15.0-163-generic debian/10 glibc/2.28 UID:GID : 0:0 netrc file : None offline mode : False


  <!-- Results of `conda info` -->

<details><summary>Command Line Output</summary>
<pre>
Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.
</pre>
</details>

<details><summary>Browser Output</summary>
<pre>
Paste the output from your browser Javascript console here.
</pre>
</details>

tlvu commented 2 years ago

Mamba was installed in the base conda env while the Jupyter is installed in another conda env. Should mamba be installed in the same conda env as the Jupyter?

fcollonval commented 2 years ago

Sorry for the late answer.

Mamba was installed in the base conda env while the Jupyter is installed in another conda env. Should mamba be installed in the same conda env as the Jupyter?

No it should not.

"error" : "RuntimeError('LockFile error. Aborting.')",

Could you try running the following command in a terminal on the machine?

/opt/conda/bin/mamba repoquery search * --json

tlvu commented 2 years ago

Could you try running the following command in a terminal on the machine?
/opt/conda/bin/mamba repoquery search * --json

@fcollonval Sorry for the late reply, from which dir I should run that command? From / I get this error:

usage: mamba [-h] [-V] command ...
mamba: error: unrecognized arguments: boot dev environment.yml etc home lib lib64 media mnt notebook_dir opt proc root run sbin srv sys tmp usr var

tlvu commented 2 years ago

Might be related, mamba install pyston_lite_autoload -c pyston -c conda-forge I get this error

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.9/site-packages/conda/exceptions.py", line 1114, in __call__
        return func(*args, **kwargs)
      File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 935, in exception_converter
        raise e
      File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 929, in exception_converter
        exit_code = _wrapped_main(*args, **kwargs)
      File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 887, in _wrapped_main
        result = do_call(args, p)
      File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 750, in do_call
        exit_code = install(args, parser, "install")
      File "/opt/conda/lib/python3.9/site-packages/mamba/mamba.py", line 497, in install
        index = load_channels(pool, channels, repos)
      File "/opt/conda/lib/python3.9/site-packages/mamba/utils.py", line 129, in load_channels
        index = get_index(
      File "/opt/conda/lib/python3.9/site-packages/mamba/utils.py", line 110, in get_index
        is_downloaded = dlist.download(api.MAMBA_DOWNLOAD_FAILFAST)
    RuntimeError: Operation not permitted: '/opt/conda/pkgs/cache/47929eba.json'

But the same using conda works: conda install pyston_lite_autoload -c pyston -c conda-forge

conda-4.13.0 mamba-0.24.0

tlvu commented 2 years ago

@fcollonval I think I got what you wanted

$ /opt/conda/bin/mamba repoquery search birdy --json


{
  "command": "/opt/conda/bin/mamba repoquery search birdy --json",
  "conda_info": {
    "GID": 1000,
    "UID": 1000,
    "active_prefix": "/opt/conda/envs/birdy",
    "active_prefix_name": "birdy",
    "av_data_dir": "/opt/conda/etc/conda",
    "av_metadata_url_base": null,
    "channels": [
      "https://repo.anaconda.com/pkgs/main/linux-64",
      "https://repo.anaconda.com/pkgs/main/noarch",
      "https://repo.anaconda.com/pkgs/r/linux-64",
      "https://repo.anaconda.com/pkgs/r/noarch"
    ],
    "conda_build_version": "not installed",
    "conda_env_version": "4.13.0",
    "conda_location": "/opt/conda/lib/python3.9/site-packages/conda",
    "conda_prefix": "/opt/conda",
    "conda_private": false,
    "conda_shlvl": 2,
    "conda_version": "4.13.0",
    "config_files": [],
    "default_prefix": "/opt/conda/envs/birdy",
    "env_vars": {
      "CIO_TEST": "<not set>",
      "CONDA_DEFAULT_ENV": "birdy",
      "CONDA_EXE": "/opt/conda/bin/conda",
      "CONDA_PREFIX": "/opt/conda/envs/birdy",
      "CONDA_PREFIX_1": "/opt/conda",
      "CONDA_PROMPT_MODIFIER": "(birdy) ",
      "CONDA_PYTHON_EXE": "/opt/conda/bin/python",
      "CONDA_ROOT": "/opt/conda",
      "CONDA_SHLVL": "2",
      "CURL_CA_BUNDLE": "<not set>",
      "PATH": "/opt/conda/envs/birdy/bin:/opt/conda/condabin:/opt/conda/envs/birdy/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "REQUESTS_CA_BUNDLE": "<not set>",
      "SSL_CERT_FILE": "<not set>"
    },
    "envs": [],
    "envs_dirs": [
      "/opt/conda/envs",
      "/notebook_dir/writable-workspace/.home/.conda/envs"
    ],
    "netrc_file": null,
    "offline": false,
    "pkgs_dirs": [
      "/opt/conda/pkgs",
      "/notebook_dir/writable-workspace/.home/.conda/pkgs"
    ],
    "platform": "linux-64",
    "python_version": "3.9.12.final.0",
    "rc_path": "/notebook_dir/writable-workspace/.home/.condarc",
    "requests_version": "2.27.1",
    "root_prefix": "/opt/conda",
    "root_writable": true,
    "site_dirs": [],
    "sys.executable": "/opt/conda/bin/python",
    "sys.prefix": "/opt/conda",
    "sys.version": "3.9.12 (main, Apr  5 2022, 06:56:58) \n[GCC 7.5.0]",
    "sys_rc_path": "/opt/conda/.condarc",
    "user_agent": "conda/4.13.0 requests/2.27.1 CPython/3.9.12 Linux/4.18.0-348.20.1.el8_5.x86_64 debian/11 glibc/2.31",
    "user_rc_path": "/notebook_dir/writable-workspace/.home/.condarc",
    "virtual_pkgs": [
      [
        "__linux",
        "4.18.0",
        "0"
      ],
      [
        "__glibc",
        "2.31",
        "0"
      ],
      [
        "__unix",
        "0",
        "0"
      ],
      [
        "__archspec",
        "1",
        "x86_64"
      ]
    ]
  },
  "error": "RuntimeError(\"Operation not permitted: '/opt/conda/pkgs/cache/3e39a7aa.json'\")",
  "exception_name": "RuntimeError",
  "exception_type": "<class 'RuntimeError'>",
  "traceback": "Traceback (most recent call last):\n  File \"/opt/conda/lib/python3.9/site-packages/conda/exceptions.py\", line 1114, in __call__\n    return func(*args, **kwargs)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 935, in exception_converter\n    raise e\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 929, in exception_converter\n    exit_code = _wrapped_main(*args, **kwargs)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 887, in _wrapped_main\n    result = do_call(args, p)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 760, in do_call\n    exit_code = repoquery(args, parser)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/mamba.py\", line 702, in repoquery\n    pool = repoquery_api.create_pool(channels, platform, use_installed)\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/repoquery.py\", line 46, in create_pool\n    load_channels(\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/utils.py\", line 129, in load_channels\n    index = get_index(\n  File \"/opt/conda/lib/python3.9/site-packages/mamba/utils.py\", line 110, in get_index\n    is_downloaded = dlist.download(api.MAMBA_DOWNLOAD_FAILFAST)\nRuntimeError: Operation not permitted: '/opt/conda/pkgs/cache/3e39a7aa.json'\n"
}

tlvu commented 2 years ago

"error": "RuntimeError(\"Operation not permitted: '/opt/conda/pkgs/cache/3e39a7aa.json'\")",

But I do have write access to that file, I tried echo >> /opt/conda/pkgs/cache/3e39a7aa.json and it works fine.

tlvu commented 2 years ago

@fcollonval I found it, the current user have to be the owner of /opt/conda/pkgs/cache/ folder. Just having write-access is not enough for mamba. I was enough for conda.

Should I still keep this issue open to fix the "current user have to be the owner of /opt/conda/pkgs/cache/ folder" to be equivalent with conda? If this won't be fixed then I can close this issue.

fcollonval commented 2 years ago

Thanks a lot @tlvu to dig that one up. Would you mind opening an issue in the mamba repository?

tlvu commented 2 years ago

Thanks a lot @tlvu to dig that one up. Would you mind opening an issue in the mamba repository?

Sure. Will close this issue once the other one in mamba repo opened.

fcollonval commented 2 years ago

Thanks

pgramme commented 1 year ago

Sure. Will close this issue once the other one in mamba repo opened.

Hi! The issue already existed in the main mamba repo: https://github.com/mamba-org/mamba/issues/488 To me it was fixed in mamba 1.2.0 (cf issue's discussion)

PS: I'm using mamba but not gator, Googling for the error first brought me here...

tlvu commented 1 year ago

Sure. Will close this issue once the other one in mamba repo opened.

Hi! The issue already existed in the main mamba repo: mamba-org/mamba#488 To me it was fixed in mamba 1.2.0 (cf issue's discussion)

PS: I'm using mamba but not gator, Googling for the error first brought me here...

Closing as per comment above. Thanks @pgramme

mamba-org / gator