ahj1592
commented
10 months ago Oh, i found the log in OUTPUT. here i post it
Closed ahj1592 closed 9 months ago
ahj1592
commented
10 months ago Oh, i found the log in OUTPUT. here i post it
celstark
commented
9 months ago FWIW, I'm having the same exact issue. My cell is print('asdf'), so not exactly fancy. The log below, up until the last line, is the connect portion. When I try to run the cell, I get the "Handle execution of cells" and we then wait with the same spinning cursors as @ahj1592. I've got others who have the same issue running Mac clients to this Linux host (Rocky-9). Other Linux hosts still work with Jupyter. Also, running a simple program as a .py file works just fine.
DonJayamanne
commented
9 months ago @celstark @ahj1592 I'm sorry you are running into this issue and thank you for filing this with the detailed logs, that helps.
Please can you install
celstark
commented
9 months ago I can't seem to get extensions to install in the insider version. It's been a bit odd on this machine that I needed to use the "always" option for the local download to get the server installed no matter what. The remote machine has wget and can use it just fine. In the end I think this wget bit is a red herring (see below), but thought I'd mention it.
(base) stark@kirby:~$ which wget
/usr/bin/wget
(base) stark@kirby:~$ wget --version
GNU Wget 1.21.1 built on linux-gnu.
-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
+ntlm +opie +psl +ssl/gnutls
...But I do see these messages in the Pty Host (Remote) output window:
2024-01-08 12:21:10.342 [warning] Shell integration cannot be enabled for executable "/bin/sh" and args ["-c","wget --version > /dev/null\nif [ $? -eq 0 ]\nthen\n\twget --connect-timeout=7 --tries=1 --dns-timeout=7 -q --header='Metadata:true' -O - http://169.254.169.254/metadata/instance?api-version=2019-03-11\nelse\n\tcurl --version > /dev/null\n\tif [ $? -eq 0 ]\n\tthen\n\t\tcurl --connect-timeout 7 -s --header='Metadata:true' http://169.254.169.254/metadata/instance?api-version=2019-03-11\n\tfi\nfi\nexit 0"]As I try to install the extension, Remote-SSH shows (Server shows the same as does the remoteagent.log file):
[14:11:48.311] [server] [14:11:48] Getting Manifest... ms-python.python
[14:11:48.441] [server] [14:11:48] Installing extension: ms-python.python
[14:11:48.682] [server] [14:11:48] Getting Manifest... ms-python.vscode-pylance
[14:11:48.735] [server] [14:11:48] Installing extension: ms-python.vscode-pylance ms-python.pythonThe extension doesn't get installed. I do get some evidence it's been downloaded:
ls -l .vscode-server-insiders/data/CachedExtensionVSIXs/
total 15650
-rw-r--r--. 1 stark starklab 16370071 Jan 8 14:11 ms-python.python-2023.23.13541005
-rw-r--r--. 1 stark starklab 144900 Jan 8 14:11 ms-python.python-2023.23.13541005.sigzipSo, in the Insider build, I can't get extensions to actually install unfortunately.
celstark
commented
9 months ago The plot thickens or gets more confusing. Before deploying the machine I'm having the issue with (kirby), I made a test machine to rough out the install as a qemu-VM. The VM really is a "draft build" of the final server. Same Rocky-9, same Warewulf 4, same Anaconda install, same -- well, everything. For grins, I decided to, in the same VSCode 1.85.1 box connect to it. There were no problems at all. So, multiple machines can't connect to my real target, but one that I can use to connect to the draft VM (that fails on the main) connects to that just fine. So, 1.85.1 at least isn't the issue.
I then made a totally clean account on kirby, disabled all my Preferences on the local machine and tried this way, wondering if some setting were triggering the bad behavior. No change. It behaves exactly the same and fails in all the same ways. So, something about kirby's setup + VSCode is the issue.
The only big difference is that on kirby /home is an NFS mount whereas the VM has it as a local folder. So, on that test user, I changed the home folder to /home/testuser. Everything works perfectly. I mean everything. No need for "local downloads", no issue with flock, no issue with Jupyter. The entire suite of problems got fixed by pulling /home off of NFS.
Now, I'd tried checking the "Lockfiles in /tmp" already with no joy there. I'd gone through systematically every option and many pairs of options in the clean test account trying to see what remote-ssh options might get things to behave better and I came up empty. To get anything to install, I needed the local download set to "always" and I needed to disable flock. With those, remote development seemed to work except for the Jupyter server. Having /home outside of NFS -- none of that needed and it worked out of the box.
The trouble of course is that this is not just a single workstation setup and having /home on the NFS share isn't something trivial to change.
Edit: Changing the nfs mount options in /etc/fstab to include proto=tcp,nfsvers=3,local_lock=flock gets it so that the remote-ssh flock option can be used and 1.85.1 can use all default options for installing the server and extensions just fine. But, I still have the issue that the Jupyter server won't run the cell.
ahj1592
commented
9 months ago @DonJayamanne Thank you for your reply, but this issue was just resolved today and I don't know why or how.
FYI, Here is my timeline:
.vscode-server, i can access to remote server. But jupyter kernel do not work. Running .py is fine in terminalHere is my log
i hope this additional log would be helpful
celstark
commented
9 months ago @ahj1592 Can you do me a favor? Can you, on your remote machine, cat /etc/fstab and cat /etc/mtab? You and I must be from a parallel universe as it's clear as day to me that you're an MRI researcher, likely a cognitive neuroscientist. Oh, and my system has been having GUI problems as well.
I ask because I just changed, after my edit above, the "local_lock" to "all" rather than just flock. And I'm now working too. My GUI apps started working as well.
ahj1592
commented
9 months ago @celstark Yes, my research is in fMRI, but I'm not a doctor (my colleagues are). local_lock=none in my case
See below image and log
cat /etc/fstab
cat /etc/mtab
rootfs / rootfs rw 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=98236560k,nr_inodes=24559140,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_prio,net_cls 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
/dev/sda1 / ext4 rw,relatime,stripe=64,data=ordered 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15262 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
[IP_ADDR]:/engrid/enslurm /engrid/enslurm nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u3 /u3 nfs4 rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=[IP_ADDR],local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u4 /home nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u5 /u5 nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u7 /u7 nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/APP /APP nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u6 /u6 nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u2 /u2 nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
[IP_ADDR]:/u2home /u2home nfs rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=[IP_ADDR],mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=[IP_ADDR] 0 0
tmpfs /run/user/1028 tmpfs rw,nosuid,nodev,relatime,size=19650504k,mode=700,uid=1028,gid=1001 0 0i masked the ip addresses
celstark
commented
9 months ago Thanks - interestingly, it's not doing anything with locking there -- at least not via NFS. It's certainly possible that they're doing something on the machine that's exporting /home (/u4).
ahj1592
commented
9 months ago @celstark Did you solve the issue(including GUI) by local_lock=all? if solved, i change the ticket to close.
DonJayamanne
commented
9 months ago Thank you for your reply, but this issue was just resolved today
closing as fixed
Does this issue occur when all extensions are disabled?: Yes
0ee08df0cf4527e40edc9aa28f4b5bd38bbff2b2Date: 2023-12-13T09:49:37.021Z Electron: 25.9.7 ElectronBuildId: 25551756 Chromium: 114.0.5735.289 Node.js: 18.15.0 V8: 11.4.183.29-electron.0 OS: Windows_NT x64 10.0.22000Steps to Reproduce:
Settings> Jupyter > Logging: Level = verbose (and reload the VS code)
make new ipynb file
select kernel
write a simple code and run a single cell
The cell do not run. No output of cells
[PROBLEMS] No problems have been detected in the workspace.
[OUTPUT] Nothing
[JUPYTER] No variables defined
Here are some related versions of extensions. all extensions are enabled in 'SSH:[MY_SERVER]'
Jupyterv2023.11.1003402403Pythonv2023.22.1Pylancev2023.12.1conda23.1.0Python3.8.17Is there anything to do? Note that i already set logger as
verbose, but i can't see any log.