R kernel not found because IRkernel package is not loaded

[gaow]

From one of our users:

I was having a new error never saw before in https://44.222.241.133/#/opcenter/jobs/s6xcnt1jw0dscgzfqapdh. Enssentially I cannot connect to R kernel in notebook but was able to connect like half an hour ago. In the stderr file. it said there is no IRkernel package but it was installed alright.

[2024-10-08 18:45:47.538] deliver:  ["Notification", "Message", "R6"]
[2024-10-08 18:45:47.539] method:  textDocument/publishDiagnostics
[2024-10-08 18:45:47.907] diagnostics task timing: 4.19669413566589   file:///home/ubuntu/.virtual_documents/hs3163/GIT/xqtl-paper-archived-10-07/AD_targets/BIN1/07_epigenomic_QTL_and_their_target_regions.nbconvert.ipynb
[I 2024-10-08 18:45:49.487 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (2/5), new random ports
Error in loadNamespace(x) : there is no package called 'IRkernel'
Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution halted
[I 2024-10-08 18:45:52.505 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (3/5), new random ports
Error in loadNamespace(x) : there is no package called 'IRkernel'
Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution halted
[I 2024-10-08 18:45:55.524 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (4/5), new random ports
Error in loadNamespace(x) : there is no package called 'IRkernel'
Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution halted
[I 2024-10-08 18:45:58.542 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (5/5), new random ports
Error in loadNamespace(x) : there is no package called 'IRkernel'
Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution halted
[W 2024-10-08 18:46:01.561 ServerApp] AsyncIOLoopKernelRestarter: restart failed
[W 2024-10-08 18:46:01.561 ServerApp] Kernel aaa422a3-8ba8-40c8-95b3-edd8263256c8 died, removing from map.
[2024-10-08 18:46:14.010] received:  Content-Length: 281
[2024-10-08 18:46:14.020] received payload.
[2024-10-08 18:46:14.020] handling request:  textDocument/hover
[2024-10-08 18:46:14.021] token: {

$ jupyter kernelspec list
Available kernels:
  bash            /home/ubuntu/.local/share/jupyter/kernels/bash
  calysto_bash    /home/ubuntu/.local/share/jupyter/kernels/calysto_bash
  ir              /home/ubuntu/.local/share/jupyter/kernels/ir
  markdown        /home/ubuntu/.local/share/jupyter/kernels/markdown
  python3         /home/ubuntu/.local/share/jupyter/kernels/python3
  sos             /home/ubuntu/.local/share/jupyter/kernels/sos

$ R

R version 4.3.3 (2024-02-29) -- "Angel Food Cake"
Copyright (C) 2024 The R Foundation for Statistical Computing
Platform: x86_64-conda-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library("IRkernel")
> 
> 

[gaow]

Seems like we are not alone here:

https://github.com/IRkernel/IRkernel/issues/583#issuecomment-1511529596

[hsun3163]

Once we run IRkernel::installspec() in a terminal R session. The issue seems to be fixed.

[Ashley-Tung]

Sounds good, then is just running IRkernel::installspec() enough to consider this resolved?

[hsun3163]

it is best that it dont happened...since it never happened to me in the past couple years when I used irkernel extensively

[Ashley-Tung]

Is there some env variable in R that can help set a library path? Even if it did work on your local computer, the method to install r kernels is different in this setup

[hsun3163]

I didnt run anything on my local computer. I run the install::IRkernel in the terminal of the cloud. Also I dont think this is a library path problem.

[Ashley-Tung]

No as in, the "past couple of years" when you used ikernel, I assumed this previous experience was done in your local machine, which is why this wasn't a problem for you before.

As for me, I have no experience with jupyter and R libraries compared to you, but a suggested solution is here: https://github.com/IRkernel/IRkernel/issues/583#issuecomment-461838492

It looks like on your container, R is installed under /home/ubuntu/.pixi/bin/R but ir is installed under /home/ubuntu/.local/share/jupyter/kernels/ir? Would this be an issue?

[hsun3163]

No as in, the "past couple of years" when you used ikernel, I assumed this previous experience was done in your local machine, which is why this wasn't a problem for you before.

No I mean use IRkernel in a cloud enviorment such as HPC and ours.

[Ashley-Tung]

And in that HPC setup, you used the same pixi-mamba.sh script to install? Because the script has gone through some changes in the past week or so

[hsun3163]

What bug me the most is that, it works for a good hours and such, but then suddenly it doesnt work anymore, with no migration .etc going on. Asking people to ready to re add the kernelspec by running IRkernel::installspec() doesnt seems to be a good long run solution.

[Ashley-Tung]

When you say "suddenly it doesnt work anymore," can you be more specific? Does the job error out, does the kernel die, or both? And when you say "suddenly it doesnt work anymore," the IRkernel WAS working, but then it didn't?

[hsun3163]

It means 20 min after I open up a notebook "A" and connect to the irkernel successfully. I opened another notebook "B" and it dont connect, then I create a new notebook "C" and it dont connect. Lastly, I disconnect the kernel from "A" and try to reconnect and it still fails.

In all three cases it give the same error msg in the stderr file as I posted.

[gaow]

To further investigate, I would look into implementation details of IRkernel::installspec() see what it actually do, and figure out the line that mattered.

[Ashley-Tung]

Going off this source: https://github.com/IRkernel/IRkernel, it allows Jupyter to see see the newly installed R kernel by installing a kernel spec for the current user.

Calling it creates a kernel.json file that Jupyter uses to recognize and interact with the R kernel. We can see it here

ubuntu@71ca067433ac:/data/interactive_analysis/hs3163/GIT/xqtl-paper/AD_targets/BIN1$ cat /home/ubuntu/.local/share/jupyter/kernels/ir/kernel.json
{
  "argv": ["/home/ubuntu/.pixi/envs/r-base/lib/R/bin/R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}"],
  "display_name": "R",
  "language": "R"
}

However, if there are multiple versions of R installed, the installspec() command might choose the wrong R version to run with Jupyter?

Is this R kernel here not the one we are looking for, or not exactly?

[gaow]

It turns out this is an issue specific to when the main kernel used is sos kernel. If we start the notebook with a pure R notebook it seems to always work.

This is also not a unique issue to MMCloud. On HPC our Jupyter Lab is 3.* for most people but when upgraded to also 4.0 as on MMC currently, this issue also occurs.

My theory for now is that sos kernel does not play well with Jupyter Lab 4.0 due to a lack of careful maintenance.

@Ashley-Tung since this is not a direct MMC issue, let's leave the issue for now, as long as people are doing well on a pure R kernel notebook? We will report if it also create issues on R kernel notebooks.

[Ashley-Tung]

Sounds good, thanks for letting me know

[hsun3163]

To clarify, though, for me, the problem occurs in a pure R notebook. But not a SOS notebook. Next time this happened, I will try to connect to SOS and then reconnect to see if that works.