Closed ahj1592 closed 9 months ago
Oh, i found the log in OUTPUT. here i post it
FWIW, I'm having the same exact issue. My cell is print('asdf')
, so not exactly fancy. The log below, up until the last line, is the connect portion. When I try to run the cell, I get the "Handle execution of cells" and we then wait with the same spinning cursors as @ahj1592. I've got others who have the same issue running Mac clients to this Linux host (Rocky-9). Other Linux hosts still work with Jupyter. Also, running a simple program as a .py
file works just fine.
@celstark @ahj1592 I'm sorry you are running into this issue and thank you for filing this with the detailed logs, that helps.
Please can you install
I can't seem to get extensions to install in the insider version. It's been a bit odd on this machine that I needed to use the "always" option for the local download to get the server installed no matter what. The remote machine has wget and can use it just fine. In the end I think this wget
bit is a red herring (see below), but thought I'd mention it.
(base) stark@kirby:~$ which wget
/usr/bin/wget
(base) stark@kirby:~$ wget --version
GNU Wget 1.21.1 built on linux-gnu.
-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
+ntlm +opie +psl +ssl/gnutls
...
But I do see these messages in the Pty Host (Remote) output window:
2024-01-08 12:21:10.342 [warning] Shell integration cannot be enabled for executable "/bin/sh" and args ["-c","wget --version > /dev/null\nif [ $? -eq 0 ]\nthen\n\twget --connect-timeout=7 --tries=1 --dns-timeout=7 -q --header='Metadata:true' -O - http://169.254.169.254/metadata/instance?api-version=2019-03-11\nelse\n\tcurl --version > /dev/null\n\tif [ $? -eq 0 ]\n\tthen\n\t\tcurl --connect-timeout 7 -s --header='Metadata:true' http://169.254.169.254/metadata/instance?api-version=2019-03-11\n\tfi\nfi\nexit 0"]
As I try to install the extension, Remote-SSH shows (Server shows the same as does the remoteagent.log
file):
[14:11:48.311] [server] [14:11:48] Getting Manifest... ms-python.python
[14:11:48.441] [server] [14:11:48] Installing extension: ms-python.python
[14:11:48.682] [server] [14:11:48] Getting Manifest... ms-python.vscode-pylance
[14:11:48.735] [server] [14:11:48] Installing extension: ms-python.vscode-pylance ms-python.python
The extension doesn't get installed. I do get some evidence it's been downloaded:
ls -l .vscode-server-insiders/data/CachedExtensionVSIXs/
total 15650
-rw-r--r--. 1 stark starklab 16370071 Jan 8 14:11 ms-python.python-2023.23.13541005
-rw-r--r--. 1 stark starklab 144900 Jan 8 14:11 ms-python.python-2023.23.13541005.sigzip
So, in the Insider build, I can't get extensions to actually install unfortunately.
The plot thickens or gets more confusing. Before deploying the machine I'm having the issue with (kirby), I made a test machine to rough out the install as a qemu-VM. The VM really is a "draft build" of the final server. Same Rocky-9, same Warewulf 4, same Anaconda install, same -- well, everything. For grins, I decided to, in the same VSCode 1.85.1 box connect to it. There were no problems at all. So, multiple machines can't connect to my real target, but one that I can use to connect to the draft VM (that fails on the main) connects to that just fine. So, 1.85.1 at least isn't the issue.
I then made a totally clean account on kirby, disabled all my Preferences on the local machine and tried this way, wondering if some setting were triggering the bad behavior. No change. It behaves exactly the same and fails in all the same ways. So, something about kirby's setup + VSCode is the issue.
The only big difference is that on kirby /home
is an NFS mount whereas the VM has it as a local folder. So, on that test user, I changed the home folder to /home/testuser
. Everything works perfectly. I mean everything. No need for "local downloads", no issue with flock, no issue with Jupyter. The entire suite of problems got fixed by pulling /home
off of NFS.
Now, I'd tried checking the "Lockfiles in /tmp" already with no joy there. I'd gone through systematically every option and many pairs of options in the clean test account trying to see what remote-ssh options might get things to behave better and I came up empty. To get anything to install, I needed the local download set to "always" and I needed to disable flock. With those, remote development seemed to work except for the Jupyter server. Having /home
outside of NFS -- none of that needed and it worked out of the box.
The trouble of course is that this is not just a single workstation setup and having /home
on the NFS share isn't something trivial to change.
Edit: Changing the nfs mount options in /etc/fstab
to include proto=tcp,nfsvers=3,local_lock=flock
gets it so that the remote-ssh flock option can be used and 1.85.1 can use all default options for installing the server and extensions just fine. But, I still have the issue that the Jupyter server won't run the cell.
@DonJayamanne Thank you for your reply, but this issue was just resolved today and I don't know why or how.
FYI, Here is my timeline:
.vscode-server
, i can access to remote server. But jupyter kernel do not work. Running .py
is fine in terminalHere is my log
i hope this additional log would be helpful
@ahj1592 Can you do me a favor? Can you, on your remote machine, cat /etc/fstab
and cat /etc/mtab
? You and I must be from a parallel universe as it's clear as day to me that you're an MRI researcher, likely a cognitive neuroscientist. Oh, and my system has been having GUI problems as well.
I ask because I just changed, after my edit above, the "local_lock" to "all" rather than just flock. And I'm now working too. My GUI apps started working as well.
@celstark Yes, my research is in fMRI, but I'm not a doctor (my colleagues are). local_lock=none
in my case
See below image and log
cat /etc/fstab
cat /etc/mtab
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=98236560k,nr_inodes=24559140,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_prio,net_cls 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/sda1 / ext4 rw,relatime,stripe=64,data=ordered 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15262 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
[IP_ADDR]:/engrid/enslurm /engrid/enslurm nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u3 /u3 nfs4 rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=[IP_ADDR],local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u4 /home nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u5 /u5 nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u7 /u7 nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/APP /APP nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u6 /u6 nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u2 /u2 nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u2home /u2home nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
tmpfs /run/user/1028 tmpfs rw,nosuid,nodev,relatime,size=19650504k,mode=700,uid=1028,gid=1001 0 0
i masked the ip addresses
Thanks - interestingly, it's not doing anything with locking there -- at least not via NFS. It's certainly possible that they're doing something on the machine that's exporting /home (/u4).
@celstark Did you solve the issue(including GUI) by local_lock=all
? if solved, i change the ticket to close.
Thank you for your reply, but this issue was just resolved today
closing as fixed
Does this issue occur when all extensions are disabled?: Yes
0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2
Date: 2023-12-13T09:49:37.021Z Electron: 25.9.7 ElectronBuildId: 25551756 Chromium: 114.0.5735.289 Node.js: 18.15.0 V8: 11.4.183.29-electron.0 OS: Windows_NT x64 10.0.22000Steps to Reproduce:
Settings> Jupyter > Logging: Level = verbose (and reload the VS code)
make new ipynb file
select kernel
write a simple code and run a single cell
The cell do not run. No output of cells
[PROBLEMS] No problems have been detected in the workspace.
[OUTPUT] Nothing
[JUPYTER] No variables defined
Here are some related versions of extensions. all extensions are enabled in 'SSH:[MY_SERVER]'
Jupyter
v2023.11.1003402403Python
v2023.22.1Pylance
v2023.12.1conda
23.1.0Python
3.8.17Is there anything to do? Note that i already set logger as
verbose
, but i can't see any log.