Visual Studio 2019 BuildSolution Hangs after CleanSolution in Jenkins

[anthonyjacques20]

Describe the bug When trying to run TcUnit-Runner via Jenkins, TcUnit-Runner hangs on vsInstance.BuildSolution. This bug appears to be fixed if vsInstance.CleanSolution is removed. There is another vsInstance.CleanSolution that causes hanging on automationInterface.StartRestartTwinCAT.

To Reproduce

1. Clone ExampleProjects
2. Have Visual Studio 2019 installed
3. Point Jenkins pipeline at ExampleProjects
4. Create Jenkins file that calls bat '"C:\\Program Files (x86)\\TcUnit-Runner\\LaunchTcUnit.bat" -t PlcTask' from within ExampleProjects/SimpleExampleProject folder.
5. Have Jenkins build off of the JenkinsFile
6. Wait for Jenkins build to hang

Expected behavior It is expected that the Visual Studio Instance is able to successfully build and start/restart TwinCAT without hanging

Screenshots Successfully running via PowerShell (completed in 90 seconds):

Getting hung in Jenkins (screenshot taken after running for about 13 minutes):

Software versions TcUnit-Runner - 0.9.3 TcUnit 1.2.0.0 Visual Studio version 2019 (16.8.30717.126) Which version of the applicable software did you use? (TcUnit-Runner, TcUnit library, TwinCAT XAE version, Visual Studio version etc)

Run environment Bare Metal

Additional context I'm able to run the LaunchTcUnit.bat file from PowerShell without any trouble. Everything works as expected but when trying to do so via Jenkins, it hangs every time. This points to there being a problem with Jenkins or Visual Studio but wanted to share in case anyone else has run into similar problems. I haven't tried this setup with XAE Shell but that is my next step, to see how this behaves in different versions of Visual Studio

[stefanbesler]

This issue also appears in TwinCat XAE 15.0 if LauchTcUnit.bat from Jenkins (Node is running as Windows Service), I „solved“ it by putting a retry(3) block around the bat command and utilizing the -u timeout option. I am not very happy with this „solution“ though.

Have you tried to run the Jenkins Node in a commandline window instead of running it as a service? It is not really an option for me, but maybe it helps in your case?

[stefanbesler]

I ended up forking the Repo and uncommenting the CleanSolution + BuildSolution code sections as I do not need them for my use case. This solved all issues that I experienced before (timeouts, hang ups…). Before uncommenting, I tried a bit around with saving the solution before Cleaning and stuff like this, but that didn’t work out.

@sagatowski Any interest in a patch that provides a switch to disable this code sections per commandline? I can happily implement this, but understand if you do not want any more switches for TcUnit-Runner + this option might be a be contraverse. You never run into this issue?

[stefanbesler]

@anthonyjacques20 fyi I stumbled into some other problems with TcUnit-Runner using it with many (semi-autogenerated) tests in my struckig project. Most of the time TcUnit-Runner was complaining about TC_STATISTICS being transmitted out of order... I ended up using the option (ParameterList) to write the result directly from the PLC.

[sagatowski]

@anthonyjacques20 & @stefanbesler, first of thanks for the very good fault description. Sorry for late response, for some reason I don't get e-mail notifications for this project...

I have not been running TcUnit-Runner with VS2019, only TcXaeShell never seen the problem there. I think most of the users of TcUnit-Runner are using the vanilla TcXaeShell.

But I remember that there was some other problem related to the cleansolution + buildsolution, and the current solution is where it ended up at (see this: https://github.com/tcunit/TcUnit-Runner/issues/5). The problem is of course that if these two always get stuck, I don't know what to do about it as it's outside of TcUnit-Runner.

Something that might be worth trying is to replace the TwinCAT Automation interface DLL from 3.2 to 3.3 and see if that makes any difference for VS2019. TcUnit-Runner is using 3.2 as it needs to support both 4020 and 4022, and I think 3.3 was introduced together with 4024 (and thus requires 4024.x installed).

Also, maybe contacting Beckhoff and asking for advice might make sense here, as it's probably not only TcUnit-Runner that has this problem?

[stefanbesler]

@sagatowski Yes, probably it’s a bug in the Automation Interface. What I noticed is, that manipulations to tsproj or plcproj via the Interface causes VS to (sometimes) see „external modifications to files in the project“. When this happens and it additionally sees a file conflict a message box pops up (no matter if using the silent option of the Automation Interface) or not. If this happens, you are stuck until you close it programmatically. What I can’t figure out is why VS sees „external modifications“ and why this is not reproduceable reliably + this seems to pop up way more often in a Jenkins/Service context. The latter, makes this very hard to figure out … if the Automation Interface is used in a debugging session from visual studio I never have any issues with it.

[sagatowski]

@stefanbesler Yes I've noticed this as well, it feels like it started to happen more often in the more recent 4024 releases. For me this however happens when I use TwinCAT normally, then this popup shows up randomly.

[sagatowski]

Note: Commit 3379fe8 might have improved this a bit.