Kernel picker doesn't find venv installed ipykernels when setting 'defaultInterpreterPath'

[casassg]

Applies To

• [X] Notebooks (.ipynb files)
• [ ] Interactive Window and\/or Cell Scripts (.py files with #%% markers)

What happened?

Installed kernels under venv don't show up in kernel pick list even after selecting venv as the main python interpreter.

jupyter kernelspec list
Available kernels:
  python3                             /Users/username/workspace/repo/.vscode/src_python_twitter_ml_tfx__tfx/venv/share/jupyter/kernels/python3
  src_python_twitter_ml_tfx__tfx      /Users/gcasassaez/workspace/repo/.vscode/src_python_twitter_ml_tfx__tfx/venv/share/jupyter/kernels/src_python_twitter_ml_tfx__tfx

However src_python_twitter_ml_tfx__tfx doesn't appear within the list in the kernel picker.

VS Code Version

Version: 1.66.2 Commit: dfd34e8260c270da74b5c2d86d61aee4b6d56977 Date: 2022-04-11T07:49:20.994Z Electron: 17.2.0 Chromium: 98.0.4758.109 Node.js: 16.13.0 V8: 9.8.177.11-electron.0 OS: Darwin x64 21.3.0

Jupyter Extension Version

v2022.3.1000901801

Jupyter logs

Visual Studio Code (1.66.2, undefined, desktop)
Jupyter Extension Version: 2022.3.1000901801.
Python Extension Version: 2022.4.1.
Multiple Workspace folders opened ~/workspace/source3/src/python, ~/workspace/source3
Info 17:29:00: ZMQ install verified.
User belongs to experiment group 'jupyterTestcf'
User belongs to experiment group 'jupyterEnhancedDataViewer'
Info 17:29:00: Old kernelspecs (created by Jupyter extension) stored in directory /Users/gcasassaez/Library/Jupyter/kernels/__old_vscode_kernelspecs
Info 17:29:01: Experiment status for python is {"enabled":true,"optInto":[],"optOutFrom":[]}
Info 17:29:01: Creating controller for jupyter-notebook with interpreter ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python
Info 17:29:01: Find preferred kernel for ~/workspace/source3/src/python/twitter/ml/tfx/examples/nb/quickstart.ipynb with metadata {"interpreter":{"hash":"2a20e6adaa8bffcd05bfe6ddb269ab947538bbfcb1d8ac9fca255e704b4bda27"},"kernelspec":{"display_name":"Python 3.7.9 64-bit ('venv': venv)","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.9"},"metadata":{"interpreter":{"hash":"2a20e6adaa8bffcd05bfe6ddb269ab947538bbfcb1d8ac9fca255e704b4bda27"}},"mldash_entity":{"created_at_millis":1621969349661,"hash":"0381c815889b5b803752fa503e425d6e7cd13a2d","inferred_pdp_safe":false,"is_vfs_dir":false,"marked_pdp_safe":false,"owner":"ddavydov","shared_to_everyone":false,"shared_to_ldap_groups":[],"shared_to_ldap_users":[],"size":38240,"tags":[],"uuid":"1397266841681088513","vfs_path":"/user/ddavydov/AIPT Docs/quickstart.ipynb"},"papermill":{"duration":407.767125,"end_time":"2020-09-24T18:04:03.852847","environment_variables":{},"exception":null,"input_path":"ns:///user/cortex/.quickstart-a3_1cb48a2aedec04631626dd60a51daa49983602d7.ipynb","output_path":"ns:///user/anonymous/.quickstart-a3_1cb48a2aedec04631626dd60a51daa49983602d7_2020-09-24T17_57_09.ipynb","parameters":{},"start_time":"2020-09-24T17:57:16.085722","version":"2.1.1"},"record_timing":true} & preferred interpreter ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python
Info 17:29:01: findKernel found Python 3.9.10 64-bit
Info 17:29:01: PreferredConnection: .jvsc74a57bd0aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49./usr/local/bin/python3./usr/local/bin/python3.-m#ipykernel_launcher found for NotebookDocument: ~/workspace/source3/src/python/twitter/ml/tfx/examples/nb/quickstart.ipynb
Info 17:29:01: TargetController found ID: .jvsc74a57bd0aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49./usr/local/bin/python3./usr/local/bin/python3.-m#ipykernel_launcher for document ~/workspace/source3/src/python/twitter/ml/tfx/examples/nb/quickstart.ipynb
Info 17:29:01: Setting controller affinity for ~/workspace/source3/src/python/twitter/ml/tfx/examples/nb/quickstart.ipynb .jvsc74a57bd0aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49./usr/local/bin/python3./usr/local/bin/python3.-m#ipykernel_launcher
Info 17:29:01: Getting activation commands for /usr/local/bin/python3 are not cached. May take a while.
Info 17:29:01: Process Execution: > ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python -m pip list
> ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python -m pip list
Info 17:29:02: Setting setActiveController for ~/workspace/source3/src/python/twitter/ml/tfx/examples/nb/quickstart.ipynb
Info 17:29:02: KernelProvider switched kernel to id = .jvsc74a57bd014f2b09c6503f1cb0a20977a1ae94d6c872dee03042d416d44ea707538c50c60./Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/python./Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/python.-m#ipykernel_launcher
Info 17:29:02: Starting Notebook in kernel.ts id = .jvsc74a57bd014f2b09c6503f1cb0a20977a1ae94d6c872dee03042d416d44ea707538c50c60./Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/python./Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/python.-m#ipykernel_launcher for ~/workspace/source3/src/python/twitter/ml/tfx/examples/nb/quickstart.ipynb
Info 17:29:02: Creating raw notebook for ~/workspace/source3/src/python/twitter/ml/tfx/examples/nb/quickstart.ipynb
Info 17:29:02: Getting preferred kernel for ~/workspace/source3/src/python/twitter/ml/tfx/examples/nb/quickstart.ipynb
Info 17:29:02: Computing working directory ~/workspace/source3/src/python/twitter/ml/tfx/examples/nb/quickstart.ipynb
Info 17:29:02: Starting raw kernel Python 3.7.9 64-bit for interpreter ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python
Info 17:29:02: Kernel launching with ports 9000,9001,9002,9003,9004. Start port is 9000
Info 17:29:02: Registering dummy command feature
Info 17:29:02: Process Execution: > ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
> ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
Info 17:29:02: Adding env Variable PYTHONNOUSERSITE to ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python
Info 17:29:02: Process Execution: > ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python -m ipykernel_launcher --ip=127.0.0.1 --stdin=9003 --control=9001 --hb=9000 --Session.signature_scheme="hmac-sha256" --Session.key=b"79abd7a3-ebd8-46ae-a9b3-b1ef0b20d76a" --shell=9002 --transport="tcp" --iopub=9004 --f=/var/folders/6w/jgs_lvqj0cd3mc418q5b193r0000gp/T/tmp-97547tJta2MNMVJa8.json
> ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python -m ipykernel_launcher --ip=127.0.0.1 --stdin=9003 --control=9001 --hb=9000 --Session.signature_scheme="hmac-sha256" --Session.key=b"79abd7a3-ebd8-46ae-a9b3-b1ef0b20d76a" --shell=9002 --transport="tcp" --iopub=9004 --f=/var/folders/6w/jgs_lvqj0cd3mc418q5b193r0000gp/T/tmp-97547tJta2MNMVJa8.json
Info 17:29:02: Process Execution: cwd: ~/workspace/source3/src/python/twitter/ml/tfx/examples/nb
cwd: ~/workspace/source3/src/python/twitter/ml/tfx/examples/nb
Info 17:29:03: ipykernel version 6.13.0 for ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python
Info 17:29:03: ipykernel location ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/lib/python3.7/site-packages/ipykernel/__init__.py for ~/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python
Info 17:29:03: Process Execution: > /usr/local/bin/python3 -m pip list
> /usr/local/bin/python3 -m pip list
Warn 17:29:05: StdErr from Kernel Process /Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/lib/python3.7/site-packages/traitlets/traitlets.py:2205: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  FutureWarning)
/Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/lib/python3.7/site-packages/traitlets/traitlets.py:2160: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use '79abd7a3-ebd8-46ae-a9b3-b1ef0b20d76a' instead of 'b"79abd7a3-ebd8-46ae-a9b3-b1ef0b20d76a"'.
  FutureWarning)

Info 17:29:05: Kernel Output: NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.

To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.

To read more about this, see https://github.com/ipython/ipython/issues/2049

To connect another client to this kernel, use:
    --existing /var/folders/6w/jgs_lvqj0cd3mc418q5b193r0000gp/T/tmp-97547tJta2MNMVJa8.json

Info 17:29:05: Raw session started and connected
Started kernel Python 3.7.9 64-bit
Info 17:29:05: Finished connecting c69e3ee4-cdc4-4bad-8791-9915c341c596
Info 17:29:05: UpdateWorkingDirectoryAndPath in Kernel
Info 17:29:05: Executing silently Code (idle) = import site\nsite.addsitedir(site.getusersitepackages())\nimport os\nimport sys\n%cd "/Users/gcasassaez/
Info 17:29:05: Executing silently Code (completed) = import site\nsite.addsitedir(site.getusersitepackages())\nimport os\nimport sys\n%cd "/Users/gcasassaez/
Info 17:29:05: Waiting for idle on (kernel): 8da1de87-84e8-40f8-b5f4-1452246286fe -> idle
Info 17:29:05: Finished waiting for idle on (kernel): 8da1de87-84e8-40f8-b5f4-1452246286fe -> idle

Coding Language and Runtime Version

Python 3.7

Language Extension Version (if applicable)

No response

Anaconda Version (if applicable)

No response

Running Jupyter locally or remotely?

Local

[rchiodo]

Thanks for the issue. Can you dump the json for the kernelspecs?

jupyter kernelspec list --json.

[casassg]

{
  "kernelspecs": {
    "python3": {
      "resource_dir": "/Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/share/jupyter/kernels/python3",
      "spec": {
        "argv": [
          "python",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3 (ipykernel)",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {
          "debugger": true
        }
      }
    },
    "test": {
      "resource_dir": "/Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/share/jupyter/kernels/test",
      "spec": {
        "argv": [
          "/Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "test",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {
          "debugger": true
        }
      }
    },
    "src_python_twitter_ml_tfx__tfx": {
      "resource_dir": "/Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/share/jupyter/kernels/src_python_twitter_ml_tfx__tfx",
      "spec": {
        "argv": [
          "/Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {
          "PYTHONPATH": "/Users/gcasassaez/workspace/source3/tss/python-shared/src/python:/Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/gen:/Users/gcasassaez/workspace/source3/cortex/notebook/src/python:/Users/gcasassaez/workspace/source3/science/packer:/Users/gcasassaez/workspace/source3/cortex/ml-metastore/src/main/python:/Users/gcasassaez/workspace/source3/src/python:/Users/gcasassaez/workspace/source3/pulse/src/python:/Users/gcasassaez/workspace/source3/cortex/pants2wheel/src/python:/Users/gcasassaez/workspace/source3/buildstats/toolinsights/src/python"
        },
        "display_name": "src_python_twitter_ml_tfx__tfx",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {
          "debugger": true
        }
      }
    },
    "088e44670129454b87f024c7182ff90d": {
      "resource_dir": "/usr/local/share/jupyter/kernels/088e44670129454b87f024c7182ff90d",
      "spec": {
        "argv": [
          "/Users/gcasassaez/workspace/bazel-sandbox/env/bin/python3",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "env2": {
      "resource_dir": "/usr/local/share/jupyter/kernels/env2",
      "spec": {
        "argv": [
          "/Users/gcasassaez/workspace/local_envs/tfma_026/env/bin/python",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "env2",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "cbd7833cce3a4ae6ae6bd62e640cc904": {
      "resource_dir": "/usr/local/share/jupyter/kernels/cbd7833cce3a4ae6ae6bd62e640cc904",
      "spec": {
        "argv": [
          "/Users/gcasassaez/workspace/bazel-sandbox/env/bin/python3",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "bb085bb0431c4c7cbce9ec350b4c88d2": {
      "resource_dir": "/usr/local/share/jupyter/kernels/bb085bb0431c4c7cbce9ec350b4c88d2",
      "spec": {
        "argv": [
          "/Users/gcasassaez/.pyenv/versions/ee-3.7/bin/python3",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "8a180438ff1d424b956b0afe31930fb9": {
      "resource_dir": "/usr/local/share/jupyter/kernels/8a180438ff1d424b956b0afe31930fb9",
      "spec": {
        "argv": [
          "/Users/gcasassaez/.pyenv/versions/ee-3.7/bin/python3",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "f2fad5a9d3e14632b1217f269cae428c": {
      "resource_dir": "/usr/local/share/jupyter/kernels/f2fad5a9d3e14632b1217f269cae428c",
      "spec": {
        "argv": [
          "/Users/gcasassaez/workspace/bazel-sandbox/env/bin/python3",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "526b351777d1412b947e5e1812bcccb8": {
      "resource_dir": "/usr/local/share/jupyter/kernels/526b351777d1412b947e5e1812bcccb8",
      "spec": {
        "argv": [
          "/Users/gcasassaez/workspace/bazel-sandbox/env/bin/python3",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "e8ad85d44b824aff9a88368581141b75": {
      "resource_dir": "/usr/local/share/jupyter/kernels/e8ad85d44b824aff9a88368581141b75",
      "spec": {
        "argv": [
          "/Users/gcasassaez/workspace/bazel-sandbox/env/bin/python3",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    },
    "7518e2ceacf743b792bb3748dfea3a10": {
      "resource_dir": "/usr/local/share/jupyter/kernels/7518e2ceacf743b792bb3748dfea3a10",
      "spec": {
        "argv": [
          "/Users/gcasassaez/workspace/bazel-sandbox/env/bin/python3",
          "-m",
          "ipykernel_launcher",
          "-f",
          "{connection_file}"
        ],
        "env": {},
        "display_name": "Python 3",
        "language": "python",
        "interrupt_mode": "signal",
        "metadata": {}
      }
    }
  }
}

[rchiodo]

So the suggested global matches and those kernelspecs don't differ in anything but name. Sounds like a similar issue we had that I thought we had fixed in #8950.

Yeah I can't repro myself. I created two identical kernelspecs in a virtual env and they both show up.

Can you set this to 'verbose' and post the output log again? It should list all of the kernelspecs we looked through.

For example, I ended up with this in the log:

Verbose 17:00:25: Loading kernelspec from ~\AppData\Local\Programs\Python\Python39\share\jupyter\kernels\python3\kernel.json for ~\AppData\Local\Programs\Python\Python39\python.exe
Verbose 17:00:25: Loading kernelspec from ~\AppData\Local\Programs\Python\Python39-32\share\jupyter\kernels\python3\kernel.json for ~\AppData\Local\Programs\Python\Python39-32\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\AboutToInstallConda\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\AboutToInstallConda\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\Conda37\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\Conda37\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\CustomJupyter\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\CustomJupyter\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\NoIpykernel\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\NoIpykernel\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\base_sub1\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\base_sub1\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\conda310\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\conda310\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\condaEnvTest\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\condaEnvTest\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\condaTestPath\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\condaTestPath\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\condaTestQT5\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\condaTestQT5\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\golden_scenario_env\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\golden_scenario_env\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\idom_kernel\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\idom_kernel\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\ipykernel_test\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\ipykernel_test\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\jupyterlabTest\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\jupyterlabTest\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\plotlyDev\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\plotlyDev\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\testNewEnv\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\testNewEnv\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\testNoPython11\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\testNoPython11\python.exe
Verbose 17:00:25: Loading kernelspec from ~\miniconda3\envs\testPython38\share\jupyter\kernels\python3\kernel.json for ~\miniconda3\envs\testPython38\python.exe
Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\AboutToInstall\share\jupyter\kernels\python3\kernel.json for d:\Source\Testing_3\AboutToInstall\Scripts\python.exe
Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\EmptyVenv2\share\jupyter\kernels\python3\kernel.json for d:\Source\Testing_3\EmptyVenv2\Scripts\python.exe
Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\LatestIPython\share\jupyter\kernels\python3\kernel.json for d:\Source\Testing_3\LatestIPython\Scripts\python.exe
Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\NoJupyter\share\jupyter\kernels\python3\kernel.json for d:\Source\Testing_3\NoJupyter\Scripts\python.exe
Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\NoPip\share\jupyter\kernels\python3\kernel.json for d:\Source\Testing_3\NoPip\Scripts\python.exe
Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\TestIPython\share\jupyter\kernels\python3\kernel.json for d:\Source\Testing_3\TestIPython\Scripts\python.exe
Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\TestKernelSpecs\share\jupyter\kernels\myExtraKernel\kernel.json for d:\Source\Testing_3\TestKernelSpecs\Scripts\python.exe
Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\TestKernelSpecs\share\jupyter\kernels\python3\kernel.json for d:\Source\Testing_3\TestKernelSpecs\Scripts\python.exe
Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\TestPython3103\share\jupyter\kernels\python3\kernel.json for d:\Source\Testing_3\TestPython3103\Scripts\python.exe
Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\xyz\share\jupyter\kernels\python3\kernel.json for d:\Source\Testing_3\xyz\Scripts\python.exe

Where these two entries are for the new virtual env I created:

Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\TestKernelSpecs\share\jupyter\kernels\myExtraKernel\kernel.json for d:\Source\Testing_3\TestKernelSpecs\Scripts\python.exe
Verbose 17:00:25: Loading kernelspec from d:\Source\Testing_3\TestKernelSpecs\share\jupyter\kernels\python3\kernel.json for d:\Source\Testing_3\TestKernelSpecs\Scripts\python.exe

[rchiodo]

It's also strange that your venv entry doesn't show up at all either. What's the root of the workspace?

[casassg]

The venv does show up (see the one on top, aka Suggested)

Verbose 18:16:46: Loading kernelspec from /usr/local/share/jupyter/kernels/088e44670129454b87f024c7182ff90d/kernel.json for 
Verbose 18:16:46: Loading kernelspec from /usr/local/share/jupyter/kernels/526b351777d1412b947e5e1812bcccb8/kernel.json for 
Verbose 18:16:46: Loading kernelspec from /usr/local/share/jupyter/kernels/7518e2ceacf743b792bb3748dfea3a10/kernel.json for 
Verbose 18:16:46: Loading kernelspec from /usr/local/share/jupyter/kernels/8a180438ff1d424b956b0afe31930fb9/kernel.json for 
Verbose 18:16:46: Loading kernelspec from /usr/local/share/jupyter/kernels/bb085bb0431c4c7cbce9ec350b4c88d2/kernel.json for 
Verbose 18:16:46: Loading kernelspec from /usr/local/share/jupyter/kernels/cbd7833cce3a4ae6ae6bd62e640cc904/kernel.json for 
Verbose 18:16:46: Loading kernelspec from /usr/local/share/jupyter/kernels/e8ad85d44b824aff9a88368581141b75/kernel.json for 
Verbose 18:16:46: Loading kernelspec from /usr/local/share/jupyter/kernels/env2/kernel.json for 
Verbose 18:16:46: Loading kernelspec from /usr/local/share/jupyter/kernels/f2fad5a9d3e14632b1217f269cae428c/kernel.json for 

[rchiodo]

That might be because it's the notebook's selected environment. What's the root of the workspace?

[casassg]

I have a double root workspace:

{
    "folders": [
        {
            "name": "Target root",
            "path": "/Users/gcasassaez/workspace/source3/src/python"
        },
        {
            "name": "Source root",
            "path": "/Users/gcasassaez/workspace/source3"
        }

[rchiodo]

The verbose output looks like it's not even looking in the venv where your custom kernels are. This might happen if you opened a folder where the venv wasn't a root folder.

That explains the problem then. It won't list venvs that aren't off of the root.

Your venv in your example is here:

/Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv

You'd have to open this folder:

/Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tf

for the venv to be found by the python extension (and for us to get it from the python extension).

[casassg]

Python extension is loading correctly the venv.

[casassg]

And it does seem to be picked up by extension

Verbose 18:16:47: Getting activation commands for /Users/gcasassaez/workspace/source3/.vscode/src_python_twitter_ml_tfx__tfx/venv/bin/python are cached.

[rchiodo]

I'm not sure what that's showing? Did you manually enter the path?

I tried creating a subdirectory with a venv and the python extension won't find that virtual environment.

Maybe you're forcing it as the active interpreter? Then you'd get the results you see here. There's only one found.

[casassg]

I'm setting the path on Python extension via python.defaultInterpreterPath on workspace file

[rchiodo]

That's likely the problem then.

Python extension doesn't know it's a virtual environment. It's just a python path. It's the 'active' interpreter when we ask the Python extension for the list so we include it as the 'suggested' kernel (which we can create just from the python path). We don't get the virtual environment properties so we don't know where to look for jupyter kernels.

Maybe we could try special casing this.

For example for a conda env, the python is at the root of the env. Whereas for a venv the python exe is in the Scripts folder (on windows).

[casassg]

I wonder if generally speaking we should just try to search for kernels under ${interpreterPath}/../share/jupyter (should cover both windows Scripts folder and linux/mac bin folder).

[casassg]

Also left https://github.com/microsoft/vscode-jupyter/discussions/9706 for an alternative where user configures these manually on the workspace file.

My use case is probs really niche (we have multi root monorepo and a small script that bootstraps a python environment + ipykernel based on bazel information for the target), but seems a good interface for folks who want to auto generate configuration for large developer bases which are okay having certain defaults set for them (my example, would benefit from having workspace defined kernel since most users will expect it to work the same way their autocomplete does which is specified already on the workspace file)

[casassg]

also, happy to contribute, but would need some pointers as Im new to VSCode extensions development

[DonJayamanne]

We don't get the virtual environment properties so we don't know where to look for jupyter kernels.

I'd say the Python extension should identify this as an interpreter and list this as active interpreter (when we call their API) which is of type venv when we (jupyter) gets the interpreter information. Else other parts of the extension could fail (e.g. installing packages and the like). Also when setting the PYTHONUSERSITE property.

@rchiodo

I.e. I don't think we (jupyter) extnsion should special case this (as the root cause seems to be in the Python extension), it should work today in Python extension, as they have the python folder and the executable (via the settings) and with that information it should be easy to determine the fact that its a venv or not.

[DonJayamanne]

Possibly related to #9718 (upstream Python ext issue)

[greazer]

@DonJayamanne, can you verify that your change above actually fixes this particular bug and then close? Thanks

[casassg]

can we wait until extension is released before closing?

[rchiodo]

can we wait until extension is released before closing?

We follow the pattern that the VS code team does. Bugs are closed when fixed by a PR. A 'verified' label is added when it's been tested to have been released in our preview bits.

[DonJayamanne]

This has been fixed in the latest pre-release version of the Jupyter extension.