Open thomers opened 1 year ago
Here's the question on SO with a bounty:
Shouldn't be an issue with running on the free of charge tier.
My guess is this broke with our Python 3.8 upgrade - do you have any way to verify? Looks like the repo you're referring to hasn't seen updates in a few years.
Hello, author of that kernel here :) found my way over from the stack overflow question.
I was taking a peek into what is going on but I'm not too familiar with the colab infrastructure. Though I can verify that I can use the kernel within that workbook linked in that top comment, but cannot change the kernel type in colab. What I mean by that is I can run the following in a cell:
!jupyter console --kernel=java --simple-prompt
and the kernel starts up properly, jupyter client can connect, and I can execute code in that kernel.
Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop starting...
Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop started.
Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop starting...
Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop started.
Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop starting...
Dec 09, 2022 7:40:21 AM io.github.spencerpark.jupyter.channels.Loop start
INFO: Loop started.
Jupyter console 6.1.0
Java 11.0.17+8-post-Ubuntu-1ubuntu218.04 :: IJava kernel 1.3.0
Protocol v5.3 implementation by jupyter-jvm-basekernel 2.3.0
In [1]: System.out.println("test");
test
So there doesn't seem to be an issue with starting the kernel in the container via jupyter but for whatever reason via colab has trouble connecting to it.
Also tried switching via getKernel().runtimeManager.changeKernelType('<session-id>', 'java')
and saw the same successful startup logs in /var/colab/app.log
:
Timestamp | Level | Message |
---|---|---|
Dec 9, 2022, 2:55:23 AM | WARNING | INFO: Loop started. |
Dec 9, 2022, 2:55:23 AM | WARNING | Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start |
Dec 9, 2022, 2:55:23 AM | WARNING | INFO: Loop starting... |
Dec 9, 2022, 2:55:23 AM | WARNING | Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start |
Dec 9, 2022, 2:55:23 AM | WARNING | INFO: Loop started. |
Dec 9, 2022, 2:55:23 AM | WARNING | Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start |
Dec 9, 2022, 2:55:23 AM | WARNING | INFO: Loop starting... |
Dec 9, 2022, 2:55:23 AM | WARNING | Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start |
Dec 9, 2022, 2:55:23 AM | WARNING | INFO: Loop started. |
Dec 9, 2022, 2:55:23 AM | WARNING | Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start |
Dec 9, 2022, 2:55:23 AM | WARNING | INFO: Loop starting... |
Dec 9, 2022, 2:55:23 AM | WARNING | Dec 09, 2022 7:55:23 AM io.github.spencerpark.jupyter.channels.Loop start |
Dec 9, 2022, 2:55:22 AM | INFO | Kernel shutdown: 25522ba0-5629-428b-a3a6-29827234a5d0 |
Dec 9, 2022, 2:55:22 AM | INFO | Kernel started: 495f9ece-6927-4283-884f-ffc29915f72c |
There are no errors and I do see the kernel connection file exists at /root/.local/share/jupyter/runtime/kernel-495f9ece-6927-4283-884f-ffc29915f72c.json
so I presume it is still running. Note that 25522ba0-5629-428b-a3a6-29827234a5d0
is the id for the default python3 kernel that got created, and in the logs it shows that as shutting down when switching to the java kernel.
Don't know if there is much else I can do to debug further on my end but happy to try anything if it will help!
@SpencerPark do you know if there's any python code invoked as part of ijava installation/setup?
In particular, this broke alongside our python 3.7 -> 3.8 transition, so I'm wondering if there's a chance some python code from ijava is getting into a bad state.
Relatedly: is there a way to get more logging from the ijava backend?
There is some python code as part of the install but after that everything is implemented in java (i.e. the argv for the kernel is java ...
). But I'm hesitant to believe the issue is there because I can use the jupyter
commands in the colab container to find and successfully start the kernel in there.
Does colab have its own way of starting kernels or is it using the same jupyter package that is user accessible in the container?
Relatedly: is there a way to get more logging from the ijava backend?
There unfortunately isn't much logging apart from some messages during startup.
Another avenue I just tried investigating though is trying to connect to the kernel that colab was creating from another tab within the same runtime via:
!jupyter console --existing=/root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971.json --simple-prompt --debug
which lead to an interesting discovery, the kernel was being started with "transport": "ipc"
rather than "tcp"
(which I knew was an option, but had never seen done and is not supported by jeromq) :
{
"shell_port": 1,
"iopub_port": 2,
"stdin_port": 3,
"control_port": 4,
"hb_port": 5,
"ip": "/root/.local/share/jupyter/runtime/kernel-78f83606-5a3f-4cf8-a89b-5db0fea63971-ipc",
"key": "",
"transport": "ipc",
"signature_scheme": "hmac-sha256",
"kernel_name": ""
}
Starting via !jupyter console --kernel=java --simple-prompt --debug
would use tcp
in the connection file and is working fine.
@craigcitro Do you know if this was another change in configuration on your side that rolled out with the 3.8 changes? And is there something we could specify in the kernel metadata to say that a specific kernel prefers tpc as the transport?
@cperry-goog and @craigcitro , any feedback or news from your side regarding
_Do you know if this was another change in configuration on your side that rolled out with the 3.8 changes?
And is there something we could specify in the kernel metadata to say that a specific kernel prefers tpc as the transport?_
Just for the record, SpencerPark created a workaround (thank you!) using a local proxy to forward the ipc to tcp channels, described here: https://stackoverflow.com/questions/74674688/google-colab-notebook-using-ijava-stuck-at-connecting-after-installation-ref
All of my notebooks with the IJava kernel that worked before now are stuck at "Connecting" after the initial IJava installation and browser page refresh.
What was working previously / expected behavior
Wait for the
Installed java kernel
messageRefresh the browser page.
Execute any cell with Java code and see the output.
Now what happens is
I can execute the first cell and get the
Installed java kernel
message, seeing the notebook status as "Connected".But after refreshing the page, the status of the notebook is stuck at "Connecting" forever,
and thus no cells with Java code can be executed.
--
I'm using Google Colab for free, but since the initial installation still works, and the notebook status is "Connected" before the page is refreshed, I assume not being a paid subscriber should not be the problem.
Any idea what has been changed, and how I can get my Java notebooks to connect again?
Example https://colab.research.google.com/github/vistec-AI/colab/blob/master/ijava.ipynb