mathworks / matlab-azure-devops-extension

Continuous Integration with MATLAB on Azure DevOps
https://marketplace.visualstudio.com/items?itemName=MathWorks.matlab-azure-devops-extension
Other
14 stars 5 forks source link

Issue with RunMatLabCommand task Version 0.10.* execution in Azure DevOps pipelines. #99

Open FarzadBTPS opened 1 year ago

FarzadBTPS commented 1 year ago

Description: We have installed the extension named MATLAB to use the task named “RunMatLabCommand” in our pipelines and have configured the task in our Azure DevOps yaml pipelines and successfully run it in the past on Virtual Machines (OS: windows). However, very rarely from when the pipeline jobs were targeted to run on Virtual Machine ScaleSets they complete the tests and show “Refreshing Simulink customizations” where the system will freeze. Past that, the system is not logging any info and failing the job as its hitting job timeout of 60 minutes. Ordinarily these jobs take ~30 minutes. The error rate is either very rare, or happens by a specific trigger as the scaleset VMs can handle dozens of jobs without problems but can then fail with timeouts, sometimes multiple in a row.

Task Details: Name: RunMatLabCommand Version : 0.10.2 Error: Generic Timeout of task past showing message “Refreshing Simulink customizations” Since when : Since we first started running Matlab on VMSS, roughly since March. At the time we were using version 0.9.19 of RunMatLabCommand. Platform : Windows OS Pool type: VMSS agent pool.

Screenshots image image

mcafaro commented 1 year ago

Hi, it is unlikely for this to be an issue with the RunMATLABCommand task itself. The more likely possibility is that Simulink is presenting a dialog box expecting user input or something in the MATLAB/Simulink process is stuck. Can you please create a service request with MathWorks technical support including these details? They can help investigate the root cause and connect with the appropriate teams to resolve the issue.

FarzadBTPS commented 1 year ago

I'll do that although I'm a bit doubtful that it's because of a dialog box. I've forgotte to take a picture so I'll do my best to describe it. At times when we've noticed that a job has gotten stuck like that, where the job isn't finished after 30 minutes and is stuck at "Refreshing Simulink customizations", it's possible to remote into the computer to see what's going on and then we only see the agent's terminal and the open Matlab instance, which shows that all tests are done. But the bigger reason I'm doubtful is that the issue can happen on the same job, doing the same thing, on the same scaleset instance. Often times a retry of the job will make it go through.

mcafaro commented 1 year ago

Makes sense. Then it sounds more likely that something else in the MATLAB/Simulink process is stuck.

You might try looking for a crash dump file, although it does not sound like MATLAB is crashing (unless an error report window is hiding behind other windows when you remote into the machine). If you have any "sl_customization.m" files on your path, you may also want to try disabling them or putting some disp commands around to see if you can pinpoint a line that might be causing the issue.

If neither of those suggestions are helpful, technical support will be in a better position to assist.

Also, if you are using MATLAB R2019a or above, you can replace the use of the RunMATLABCommand task with a direct call to MATLAB (as shown below) in your pipeline to rule out the possibility that the task itself is causing the hang. However, I would be quite surprised if you found that to be the case.

Replace:

  - task: RunMATLABCommand@0
    inputs:
      command: myscript

With:

  - script: matlab -batch "myscript"
FarzadBTPS commented 1 year ago

Great suggestions, thanks. I'll see if I can give some of them a go.