Loccret commented 1 year ago

Is there an existing issue for this bug?

[X] I have searched the existing issues

Required Troubleshooting Steps

[X] I have tried both values of the remote.SSH.useLocalServer setting
[x] I have followed these troubleshooting steps

Connect Locally

It connects successfully

->

No response

Expected Behavior

Hello,

I have noticed a little issue while running a fastai model on a remote server using Remote-SSH on Jupyter. While the fit function of fastai package correctly prints the training information on my local machine,

the same information is not printed when using Remote-SSH on Jupyter.

Thank you.

Steps To Reproduce

local enviroment: VSCODE Version: 1.79.0-insider (user setup) Commit: a7133ae9e21e01ed204b02e17e8fc3e91dcadae7 Date: 2023-05-09T09:51:22.467Z Electron: 22.5.1 Chromium: 108.0.5359.215 Node.js: 16.17.1 V8: 10.8.168.25-electron.0 OS: Windows_NT x64 10.0.19044 Sandboxed: Yes Remote-SSH version: v0.102.0
remote server: OS: Linux-5.4.0-100-generic-x86_64-with-glibc2.31 python : 3.10.10 tsai : 0.3.6 fastai : 2.7.12 fastcore : 1.5.29 torch : 2.0.0

code


from tsai.all import *
from fastai.callback.tracker import EarlyStoppingCallback
from IPython.display import clear_output

dsid = 'NATOPS' X, y, splits = get_UCR_data(dsid, return_split=False) # type: ignore X_train, y_train, X_test, y_test = get_UCR_data(dsid, return_split=True) # type: ignore X, y, splits = combine_split_data([X_train, X_test], [y_train, y_test]) tfms = [None, [Categorize()]] dsets = TSDatasets(X, y, tfms=tfms, splits=splits, inplace=True) dls = TSDataLoaders.from_dsets(dsets.train, dsets.valid, metrics = [accuracy], bs=[64, 128], batch_tfms=[TSStandardize()], num_workers=0) # type: ignore

for _ in np.arange(2):

model = InceptionTime(dls.vars, dls.c) learn = Learner(dls, model, metrics=accuracy, cbs = [# ShowGraph(), EarlyStoppingCallback(monitor='accuracy', comp=np.greater, patience=10), SaveModel(monitor='accuracy', fname = f'model', verbose=False)] )

with learn.no_bar(), learn.no_logging():

clear_output(wait = False)

learn.fit_one_cycle(20, lr_max=1e-3)


### Remote-SSH Log

<details>
<summary>Remote-SSH Log</summary>
<p>

[20:11:29.679] Log Level: 2 [20:11:29.695] SSH Resolver called for "ssh-remote+li", attempt 1 [20:11:29.696] "remote.SSH.useLocalServer": false [20:11:29.696] "remote.SSH.showLoginTerminal": false [20:11:29.697] "remote.SSH.remotePlatform": {"li":"linux"} [20:11:29.697] "remote.SSH.path": undefined [20:11:29.697] "remote.SSH.configFile": undefined [20:11:29.697] "remote.SSH.useFlock": true [20:11:29.697] "remote.SSH.lockfilesInTmp": false [20:11:29.698] "remote.SSH.localServerDownload": auto [20:11:29.698] "remote.SSH.remoteServerListenOnSocket": false [20:11:29.698] "remote.SSH.showLoginTerminal": false [20:11:29.698] "remote.SSH.defaultExtensions": [] [20:11:29.698] "remote.SSH.loglevel": 2 [20:11:29.699] "remote.SSH.enableDynamicForwarding": true [20:11:29.699] "remote.SSH.enableRemoteCommand": false [20:11:29.699] "remote.SSH.serverPickPortsFromRange": {} [20:11:29.699] "remote.SSH.serverInstallPath": {} [20:11:29.706] VS Code version: 1.79.0-insider [20:11:29.706] Remote-SSH version: remote-ssh@0.102.0 [20:11:29.706] win32 x64 [20:11:29.707] SSH Resolver called for host: li [20:11:29.708] Setting up SSH remote "li" [20:11:29.714] Using commit id "a7133ae9e21e01ed204b02e17e8fc3e91dcadae7" and quality "insider" for server [20:11:29.720] Install and start server if needed [20:11:29.725] Checking ssh with "C:\Program Files\Google\Chrome\Application\ssh.exe -V" [20:11:29.727] Got error from ssh: spawn C:\Program Files\Google\Chrome\Application\ssh.exe ENOENT [20:11:29.728] Checking ssh with "C:\Program Files (x86)\Common Files\Oracle\Java\javapath\ssh.exe -V" [20:11:29.729] Got error from ssh: spawn C:\Program Files (x86)\Common Files\Oracle\Java\javapath\ssh.exe ENOENT [20:11:29.729] Checking ssh with "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\bin\ssh.exe -V" [20:11:29.731] Got error from ssh: spawn C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\bin\ssh.exe ENOENT [20:11:29.731] Checking ssh with "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\libnvvp\ssh.exe -V" [20:11:29.732] Got error from ssh: spawn C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\libnvvp\ssh.exe ENOENT [20:11:29.732] Checking ssh with "C:\Windows\system32\ssh.exe -V" [20:11:29.734] Got error from ssh: spawn C:\Windows\system32\ssh.exe ENOENT [20:11:29.734] Checking ssh with "C:\Windows\ssh.exe -V" [20:11:29.736] Got error from ssh: spawn C:\Windows\ssh.exe ENOENT [20:11:29.736] Checking ssh with "C:\Windows\System32\Wbem\ssh.exe -V" [20:11:29.737] Got error from ssh: spawn C:\Windows\System32\Wbem\ssh.exe ENOENT [20:11:29.737] Checking ssh with "C:\Windows\System32\WindowsPowerShell\v1.0\ssh.exe -V" [20:11:29.739] Got error from ssh: spawn C:\Windows\System32\WindowsPowerShell\v1.0\ssh.exe ENOENT [20:11:29.739] Checking ssh with "C:\Windows\System32\OpenSSH\ssh.exe -V" [20:11:29.781] > OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2

[20:11:29.785] Running script with connection command: "C:\Windows\System32\OpenSSH\ssh.exe" -T -D 56717 li bash [20:11:29.790] Terminal shell path: C:\Windows\System32\cmd.exe [20:11:30.072] > ]0;C:\Windows\System32\cmd.exe [20:11:30.073] Got some output, clearing connection timeout [20:11:30.120] > li@133.55.113.219's password: [20:11:30.120] Showing password prompt [20:11:57.401] Got password response [20:11:57.402] "install" wrote data to terminal: "****" [20:11:57.414] > [20:11:57.551] > c00ce4d8c9d9: running [20:11:57.607] > Acquiring lock on /home/li/.vscode-server-insiders/bin/a7133ae9e21e01ed204b02e17

e8fc3e91dcadae7/vscode-remote-lock.li.a7133ae9e21e01ed204b02e17e8fc3e91dcadae7
[20:11:57.628] > Found existing installation at /home/li/.vscode-server-insiders/bin/a7133ae9e21e 01ed204b02e17e8fc3e91dcadae7... Checking /home/li/.vscode-server-insiders/.a7133ae9e21e01ed204b02e17e8fc3e91dcad ae7.log and /home/li/.vscode-server-insiders/.a7133ae9e21e01ed204b02e17e8fc3e91d cadae7.pid for a running server Looking for server with pid: 81075 [20:11:57.644] > Found running server...

*

Visual Studio Code Server

[20:11:57.659] > * By using the software, you agree to

the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-lice nse) and

the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacyst atement).

[20:11:57.673] > Checking server status on port 42047 with wget [20:11:57.685] > c00ce4d8c9d9: start
SSH_AUTH_SOCK====
DISPLAY==== webUiAccessToken==== listeningOn==42047== osReleaseId==ubuntu== arch==x86_64==
vscodeArch==x64== bitness==64== tmpDir==/tmp== platform==linux== unpackResult==== didLocalDownload==0== downloadTime==== installTime==== extInstallTime==== serverStartTime==== connectionToken==11111111-a11a-1a11-1a11-aaa1a11a111a== c00ce4d8c9d9: end [20:11:57.686] Received install output:
SSH_AUTH_SOCK====
DISPLAY==== webUiAccessToken==== listeningOn==42047== osReleaseId==ubuntu== arch==x86_64==
vscodeArch==x64== bitness==64== tmpDir==/tmp== platform==linux== unpackResult==== didLocalDownload==0== downloadTime==== installTime==== extInstallTime==== serverStartTime==== connectionToken==11111111-a11a-1a11-1a11-aaa1a11a111a==

[20:11:57.687] Remote server is listening on 42047 [20:11:57.687] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":42047},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/tmp","platform":"linux","connectionToken":"11111111-a11a-1a11-1a11-aaa1a11a111a"},"installUnpackCode":""} [20:11:57.692] Starting forwarding server. localPort 56728 -> socksPort 56717 -> remotePort 42047 [20:11:57.694] Forwarding server listening on 56728 [20:11:57.695] Waiting for ssh tunnel to be ready [20:11:57.698] [Forwarding server 56728] Got connection 0 [20:11:57.701] Tunneled 42047 to local port 56728 [20:11:57.701] Resolved "ssh-remote+li" to "127.0.0.1:56728" [20:11:57.712] > [20:11:57.719] ------

[20:11:57.733] [Forwarding server 56728] Got connection 1 [20:11:57.754] [Forwarding server 56728] Got connection 2 [20:11:58.421] [Forwarding server 56728] Got connection 3 [20:11:58.423] [Forwarding server 56728] Got connection 4 [20:13:24.202] [Forwarding server 56728] Got connection 5



</p>
</details>

### Anything else?

_No response_

roblourens commented 1 year ago

I doubt that it really has anything to do with SSH, do other cells print output correctly?

Loccret commented 1 year ago

Hello, @roblourens!

Thank you for taking the time to address this issue. I just wanted to let you know that the output from other cells in my notebook is printing correctly. Additionally, when running the code on my local machine, the training information from Fastai is also printing correctly.

I am not certain where the appropriate place is to raise this issue, but after verifying the versions of the Python libraries and extensions in VSCode. I decided to raise it here.

I apologize if this is an unsuitable location for the issue.

microsoft / vscode-remote-release

[Remote-SSH Bug]: Training information not printed correctly when running fastai model through Remote-SSH on Jupyter #8464