Closed JulianSchmidtkeGWS closed 2 years ago
freddydk
commented
2 years ago If I am to assist with this, I would need somebody to setup a repo, where I can have access and perform various tests and run unlimited number of pipelines, for me to repro this.
JulianSchmidtkeGWS
commented
2 years ago Hi Freddy, thank you for your Reply. I have invited you to our DevOps.
JulianSchmidtkeGWS
commented
2 years ago Freddy and i could find the cause of this and https://github.com/microsoft/azure-pipelines-tasks/issues/15819 (PSSession not being canceled in the communication to the container) and fixed it for us by setting "UsePSSession" to false in the ContainerHelper Config.
Our VM Image is now created with this script added:
Import-Module BCContainerHelper
$bcContainerHelperConfig.UsePsSession = $false
$bcContainerHelperConfig | ConvertTo-Json | Set-Content "C:\ProgramData\BcContainerHelper\BcContainerHelper.config.json"Thank you again, Freddy!
The main Problem of PSSession not working in the azure pipeline might still occur though.
freddydk
commented
2 years ago I always recomment that people ONLY set the settings they actually are changing in the settings file. This way - you force all current values into the settings file, which you might not want.
I would recommend this instead:
@{ "UsePsSession" = $false } | ConvertTo-Json | Set-Content "C:\ProgramData\BcContainerHelper\BcContainerHelper.config.json"
Import-Module BCContainerHelperThanks for the help Julian - I will write a blog post about the topic and link from various github issues to that.
freddydk
commented
2 years ago Also, next version of ContainerHelper will automatically turn off the use of PsSessions in Invoke-ScriptInBcContainer and use Docker Exec when running Azure DevOps Pipelines or GitHub Actions Workflows.
freddydk
commented
2 years ago I decided to revert this decision. As the use of docker exec does come with some performance impact, I decided to keep the behavior, where you need to add usePsSession to the config file.
JulianSchmidtkeGWS
commented
2 years ago Understandable. I have compared a few runs in our history and the performance impact does not seem too dramatic.
The biggest change is in our RunTests Task from 70-75min -> 78-80min in our biggest app. But that one tests each codeunit on its own which means that a request is sent for each codeunit (200+). Time to redesign that one on our end.
I hope the Azure Pipelines Team finds a good solution to use PSSession without the timeouts, but this works fine for us 👍
freddydk
commented
2 years ago Establishing a PsSession (first time you invoke a script in a container in that powershell session) - is around 1 second (on my machine) Subsequent invokes to that PsSession from the same PowerShell session is around 5 ms
Invoking a script using docker exec is around 0,9 second (every time)
So, if you have a lot of tasks (individual PowerShell sessions), the difference isn't big. If you use Run-AlPipeline, which does everything in one PowerShell session - the difference becomes huge (around 20% is what I saw)
This is related to https://github.com/microsoft/azure-pipelines-tasks/issues/15819 which is no longer assigned to anyone.
Describe the issue We have custom build Azure Pipelines with BCContainerhelper. About 10% of each Stage is cancled by the system, because it times out. The problem is, that the stage actually was successful but the tasks never exits. This only happens in Powershell tasks (FilePath and Inline) where BCContainerhelper is used. We use a custom vm-scaleset with bccontainerhelper and additional tools preinstalled. Other users report the same issue with normal VM_Agents.
Scripts used to create container and cause the issue This happens with our create-container, run-tests and sign-app scripts.
Example for Create Container:
YAML:
PS:
Full output of scripts
Screenshots It usually takes only a few minutes to create the container. Its not responding for 20+ min
Additional context It happens in about 10% of our pipelines for different tasks always in combination with bccontainerhelper.
Another user has created an example pipeline that might reproduce it easier: https://github.com/microsoft/azure-pipelines-tasks/issues/15819#issuecomment-1083414837
We have tried different versions of bccontainerhelper and always have the same issue.
We tried to upgrade the VM Size (Cores and RAM) and have the same Issue