search

HodorNV / ALOps

ALOps
55 stars 24 forks source link

Recompilation with ALOpsAppCompiler@2 Takes Just as Long as the Initial Compilation #729

Closed TroelsMuffThygesen closed 3 months ago

TroelsMuffThygesen commented 3 months ago

In our build pipeline we have special handling for xliff file generation which requires that we first compile the app once, then generates the xliff files and finally recompiles the app to include the new xliff files. (The recompilation is done without codeanalysers) We have until now created docker containers and done the compilation inside of this and with this approach we have an example where the initial compilation takes 30 seconds and the recompilation 20 seconds. In order to optimize identification of compilation errors we are working on a new build pipeline structure where we use the so called super compiler task, ALOpsAppCompiler@2, and with this structure it now takes 1 minute and 30 seconds for the initial compilation and 1:20 for the second compilation. We suspect that the recompilation downloads/retrieves all the artifacts once more, otherwise we would have expected the compilation time to be significantly lower and closer to the recompilation task done in docker.

So the question is. Is there some way to secure that artificats already downloaded/retrieved are not downloaded/retrieved again?

We see the same scenario when the Test app is compiled. This app is in the same repository but in a different al project folder. Here we of course need additional artifacts for the test toolkit apps, so some additional time is expected.

We set up to three tasks as follows:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

KR Troels M. Thygesen

kasperdj commented 3 months ago

..also it seams like that the super compiler always downloads the artifacts even though the have been pre-downloaded earlier on the build server

waldo1001 commented 3 months ago

Are you sue it's downloading every time?
We're working with "Weekly" in stead of "Current" just to prevent it taking every single new latest build (which happens a lot - multiples a day).

@TroelsMuffThygesen - can we have your log?

waldo1001 commented 3 months ago

In the AppCompilerV2, we're using default BCCH functionality to download artifacts, including its caching.

We tested, and retested, and once we also got an unexplainable re-download, but usually, it just takes the cache.. .

kasperdj commented 3 months ago

Here is a fresh log for you

01 - Build log example 02 - Build log example.txt

waldo1001 commented 3 months ago

Thanks.

You're suffering from multiple "Get-BCArtifactUrl" steps, which take a while.

We did some significant changes to this step, so we'll have to deploy this first, and then see what we can do further (as the changes could already impact this behavior).

waldo1001 commented 3 months ago

Please test the above in the latest release (v1.463)

kasperdj commented 3 months ago

Tested a new pipeline run after the ALOps update. "Get-BCArtifactUrl" still run for each compile step and uses approx 32 second before starting the compilation. Before the update it used 47-50 seconds before starting the compilation, so a small improvement, but still room for better caching ;-)

03 - Build test take 2 after update 04 - Build test take 2 after update.txt

waldo1001 commented 3 months ago

All steps are running in Isolation, so one compile and the other, is going to do the same steps. That's the building block design. Not sure we'll be able to make multiple compile-steps "talk to each other"..

I see it takes 24 seconds to find the weekly artifact version: image About the same on our end.

This takes 20 seconds: Get-bcartifactUrl -type sandbox -select Weekly

This takes 0.5 seconds: Get-bcartifactUrl -type sandbox -version 23.5.16502.17182

What if ... we could run a separate step that gets the exact version (with PowerShell), and use that version further in the script. It would save us another 40 seconds .. ?

kasperdj commented 3 months ago

It would be OK to have an initial seperate step that collects the needed artifacts (weekly sandbox for a given localization in our example) and that the super compiler can trust and use these artifact, if present on the build server. The Super Compiler should be able to use these artifacts in the rest of the pipeline. If the new step for downloading artifacts have not been run, then do as you do now in the Super Compiler.

PS: in an optimal world the new steps will also not download artifacts if they have already been pre-downloaded on the build server, either from a previous pipeline or via a powershell warmup script like the one we run in the night on all build servers

Catch me on Teams or call me if you need more input

waldo1001 commented 3 months ago

PS: in an optimal world the new steps will also not download artifacts if they have already been pre-download on the build server, either from a previous pipeline or via a powershell warmup script like the one we run in the night on all build servers

True - I don't know why BCCH does that from time to time. We simply use the "Get-ArtifactUrl" and the "Download-Artifacts" from BCCH (no other choice here...) - we double checked: without -force switch.
I must say, it doesn't "always" download a new version, but sometimes it does that without reason.. .

waldo1001 commented 3 months ago

Here's an example that seems to work...

- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      $url = Get-bcartifactUrl -type sandbox -select Weekly
      $version = [regex]::Match($url, "/(\d+\.\d+\.\d+\.\d+)/").Groups[1].Value

      Write-Host "##vso[task.setvariable variable=BCVersion;]$($version)"

- task: ALOpsAppCompiler@2
  inputs:
    artifacttype: 'Sandbox'
    appversiontemplate: 'A.A.*.0'
    alcodeanalyzer: None
    artifactversion: $(BCVersion)

- task: ALOpsAppCompiler@2
  inputs:
    artifacttype: 'Sandbox'
    appversiontemplate: 'A.A.*.0'
    alcodeanalyzer: None
    artifactversion: $(BCVersion)

- task: ALOpsAppCompiler@2
  inputs:
    artifacttype: 'Sandbox'
    appversiontemplate: 'A.A.*.0'
    alcodeanalyzer: None
    artifactversion: $(BCVersion)
kasperdj commented 3 months ago

Your suggestion improved the overall compilation time for our pipeline which is now down to 3m 14s compared to previously 3m 48s

image

Let's close the case, thx

Arthurvdv commented 3 months ago

What a briljant idea! I'm going to shamelessly take this and also going to apply this in our pipelines.

I'm going to take this a step further and also apply Get your BC artifact URLs without PowerShell which caches the result for an hour, so the next pipeline (within the hour) also can benefit.

It's going to be something like this:

- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      $url = Invoke-WebRequest "https://bca-url-proxy.azurewebsites.net/bca-url/Sandbox/W1?select=Daily&DoNotRedirect=true" -UseBasicParsing
      if ([string]::IsNullOrEmpty($url)) {
        $url = Get-BCArtifactUrl -type Sandbox -country 'W1' -select Daily
      }
      Write-Host $url
      $version = [regex]::Match($url, "/(\d+\.\d+\.\d+\.\d+)/").Groups[1].Value

      Write-Host "##vso[task.setvariable variable=BCVersion;]$($version)"
Arthurvdv commented 3 months ago

To make the bcartifacturl-proxy more accessible in Azure DevOps, I've created an extension for it: bcartifacturl-proxy.

It's an open-source extension, freely available for use, leveraging the instance generously provided by @tfenster for the community.