configure timeouts

[jhgoebbert]

Hello everyone,

we came across timeout-issues lately in our setup which can be seen from a screenshot below.

It seems to me as if these issues are timeouts from the function fetchWithTimeout(): It would be great we could increase these timeouts (perhaps by a factor one can set through an environment variable) for systems where certain steps can take longer that 10-15 seconds.

Best regards, Jens

P.S: I came across a very short timeout (100) which might be a bug - see below.

Timout-Screenshot:

fetchWithTimout(): https://github.com/search?q=repo%3Amathworks%2Fmatlab-proxy%20fetchWithTimeout&type=code

[prabhakk-mw]

Hi @jhgoebbert

The highlighted value in the screenshot is from a test file that we use to test these components. It shouldn't effect production workflows.

Can you confirm that MATLAB has indeed started up in the background?

Do you have any logs coming from matlab-proxy that you could share when this occurs?

If IIRC, there is a 2 minute timeout where we wait for MATLAB to respond to the proxy.

How long does MATLAB typically take to launch in this environment?

Is this issue sporadic?

[jhgoebbert]

Hello @prabhakk-mw

yes, MATLAB is running in the background even though the web-ui shows the error of timeout:

[goebbert1@jwlogin01 juwels]$ ps aux | grep  matlab 
goebber+  7390  0.0  0.0  13720  1624 pts/28   S+   16:27   0:00 grep --color=auto matlab
goebber+ 26431  0.1  0.0 247612 44268 ?        S    15:50   0:02 /p/software/juwels/stages/2023/software/Python/3.10.4-GCCcore-11.3.0/bin/python /p/software/juwels/stages/2023/software/JupyterProxy-Matlab/0.8.0-GCCcore-11.3.0-2023.3.6/bin/matlab-proxy-app --config Jupyter
goebber+ 27234  0.0  0.0  30240  4248 ?        S    15:50   0:00 /bin/bash /p/software/juwels/stages/2023/software/MATLAB/2023a-GCCcore-11.3.0/bin/matlab -nosplash -nodesktop -softwareopengl -r try; run('/p/software/juwels/stages/2023/software/JupyterProxy-Matlab/0.8.0-GCCcore-11.3.0-2023.3.6/lib/python3.10/site-packages/matlab_proxy/matlab/startup.m'); catch ME; disp(ME.message); end;
goebber+ 27259  3.8  0.3 13465076 2420736 ?    Sl   15:50   1:25 /p/software/juwels/stages/2023/software/MATLAB/2023a-GCCcore-11.3.0/bin/glnxa64/MATLAB -nosplash -softwareopengl -r try; run('/p/software/juwels/stages/2023/software/JupyterProxy-Matlab/0.8.0-GCCcore-11.3.0-2023.3.6/lib/python3.10/site-packages/matlab_proxy/matlab/startup.m'); catch ME; disp(ME.message); end; -nodesktop

I can read the following in stderr. But keep in mind from above - MATLAB is running. I can open the JupyterLab terminal (so on the same system) and start matlab on there without issue.

INFO:MATLABProxyApp:Found MATLAB Executable: /p/software/juwels/stages/2023/software/JupyterProxy-Matlab/0.8.0-GCCcore-11.3.0-2023.3.6/bin/matlab with Root: /p/software/juwels/stages/2023/software/JupyterProxy-Matlab/0.8.0-GCCcore-11.3.0-2023.3.6
INFO:MATLABProxyApp:
============================
MATLAB can be accessed at:
http://127.0.0.1:xxxxx/user/j.goebbert_at_fz-juelich.de/xxxxxxxxxxxxxxxxxxxxxxxxxx/matlab?mwi_auth_token=xxxxxxxxxxxxxxxxxx
============================

INFO:MATLABProxyApp:!!! Launching MATLAB without providing any additional licensing information. This requires MATLAB to have been activated on the machine from which its being launched !!!
INFO:MATLABProxyApp:Deleting any cached licensing files!
INFO:MATLABProxyApp:Stopped (any running) MATLAB process.

I increased this from 100 to 1800 already: https://github.com/mathworks/jupyter-matlab-proxy/blob/main/src/jupyter_matlab_proxy/__init__.py#L96

Most of the time we stay below 1 minute. But it can take sometimes much longer. This is a GPFS issue over which we have limited control and have to live with.

Yes, this is an sporadic issue. A classical timeout-problem. It works on the same system a minute later again.

Best regards Jens Henrik

[prabhakk-mw]

@jhgoebbert Please use the environment variable MWI_PROCESS_START_TIMEOUT to configure the timeout in seconds that you would like to set for your users. If this value is unset, the default value is 120 seconds.

[jhgoebbert]

Great, thank you!

[jhgoebbert]

btw: I also increased the timeout of jupyter_matlab_proxy https://github.com/mathworks/jupyter-matlab-proxy/blob/main/src/jupyter_matlab_proxy/__init__.py#L96

[prabhakk-mw]

The timeout to jupyter-matlab-proxy is meant to indicate how long matlab proxy will take to respond to http requests, and not how long it takes for MATLAB to respond. I don't think it's necessary to update this value.

More information about this value can see be read from https://jupyter-server-proxy.readthedocs.io/en/latest/server-process.html#timeout