search

kiuwan / tfs-vsts-extension

Extension for Microsoft TFS and VSTS. Adds build tasks to run Kiuwan scans
3 stars 9 forks source link

Root VSTS Agent directory is clensed after task runs #2

Closed tasadar2 closed 6 years ago

tasadar2 commented 6 years ago

Issue After running a build with the Kiuwan VSTS task, the VSTS agent involved crashes. When digging into why it crashed, we found that most of the file/folder structure for the agent has been deleted. We have attempted multiple builds to confirm, and each time the Kiuwan task is run, regardless of the build, the agent that runs the task is no longer in a usable state.

Based on the logs and the result, my guess is that the task attempts to clean the directory Kiuwan was downloaded to, but might be using one directory too high in the directory hierarchy. Namely these lines from the Kiuwan build task log

[command]C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command "Move-Item -Path 'C:\TfsData\Agent\12a2e36b-7948-4c20-b3d8-12754c3511d1\KiuwanLocalAnalyzer' -Destination 'C:\TfsData\Agent'"
Kiuwan Local Analyzer installed at: C:\TfsData\Agent!

Steps to Reproduce Queue a build with the Kiuwan VSTS build task.

Observations

Log from Kiuwan task

##[section]Starting: Kiuwan baseline analysis
==============================================================================
Task         : Kiuwan Baseline Analysis
Description  : Run a Kiuwan baseline Kiuwan analysis on the source code extracted by the build configuration.
Version      : 2.2.1
Author       : Javier Salado
Help         : WARNING: Make sure you define the Kiuwan credentials variables in the build definition KiuwanUser and KiuwanPasswd. Otherwise the analysis will fail.
==============================================================================
37c785d3-c5a1-49d7-bb01-0e7ac155a1cf exists true
Kiuwan application from Kiuwan app list: Lola.ORM
Running on Agent: server (win32)
Checking for KLA previously installed in the agent home: C:\TfsData\Agent
No KLA installation found...
Downloading and installing KLA in the agent home: C:\TfsData\Agent
Downloading: https://www.kiuwan.com/pub/analyzer/KiuwanLocalAnalyzer.zip
Extracting archive
[command]C:\Windows\system32\chcp.com 65001
Active code page: 65001
[command]C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -Sta -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command "$ErrorActionPreference = 'Stop' ; try { Add-Type -AssemblyName System.IO.Compression.FileSystem } catch { } ; [System.IO.Compression.ZipFile]::ExtractToDirectory('C:\TfsData\Agent\KiuwanLocalAnalyzer.zip', 'C:\TfsData\Agent\12a2e36b-7948-4c20-b3d8-12754c3511d1')"
[command]C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command "Move-Item -Path 'C:\TfsData\Agent\12a2e36b-7948-4c20-b3d8-12754c3511d1\KiuwanLocalAnalyzer' -Destination 'C:\TfsData\Agent'"
Kiuwan Local Analyzer installed at: C:\TfsData\Agent!
Running Kiuwan analysis
[command]C:\Windows\system32\cmd.exe /D /S /C "C:\TfsData\Agent\KiuwanLocalAnalyzer\bin\agent.cmd -n Lola.ORM -s C:\TfsData\Agent\_work\1\s -l "null LOLA.ORM_1.0.18081.13" -c -wr --user ******** --pass ******** "exclude.patterns=**/*.min.js,**/*.Designer.vb,**/*Reference.vb,**/*Service.vb,**/*Silverlight.vb,**/*.designer.vb,**/*reference.vb,**/*service.vb,**/*silverlight.vb,**/*.Designer.cs,**/*Reference.cs,**/*Service.cs,**/*Silverlight.cs,**/*.designer.cs,**/*reference.cs,**/*service.cs,**/*silverlight.cs,**/.*,**/Pods/BuildHeaders/**/*.h,**/Pods/Headers/**/*.h" "include.patterns=**/**" "include.patterns=**/*" "encoding=UTF-8" "supported.technologies=abap,actionscript,aspnet,c,cobol,cpp,csharp,html,java,javascript,jcl,jsp,natural,objectivec,oracleforms,perl,php,powerscript,python,rpg4,ruby,swift,vb6,vbnet,xml" "memory.max=1024m" "timeout=3600000" "ignore=architecture,insights""

   #       #
   #       #
   #
   #    #  #   #    #  #          #   ###    # ###
   #   #   #   #    #   #   ##   ##  #   #   ###  #
   #  #    #   #    #   #   ##   #       #   #    #
   ####    #   #    #   #  ## #  #    ####   #    #
   #  #    #   #    #    # #  # #    #   #   #    #
   #  ##   #   #    #    # #  # #   #    #   #    #
   #   #   #   #    #    ##    ##   #    #   #    #
   #    #  #   ######     #    #     #####   #    #

                                   www.kiuwan.com

java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

Current Kiuwan Local Analyzer version: master.1111.p495.q8302
New Kiuwan Local Analyzer version available: master.1111.p495.q8302
Kiuwan Local Analyzer is up to date
Launching...

java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

Created dir: C:\TfsData\Agent\KiuwanLocalAnalyzer\temp\Lola.ORM.1647684088
Deleting: C:\TfsData\Agent\KiuwanLocalAnalyzer\temp\Lola.ORM.1647684088\results\tmp.kiuwan
No analysis configuration file found. Defaults will be used.
-----------------------------------------------------
Starting analysis for application "Lola.ORM"
-----------------------------------------------------
Source paths:       C:\TfsData\Agent\_work\1\s
Analysis scope:     baseline
Change req. status: 
Branch name:        
Change request:     
Analysis label:     null LOLA.ORM_1.0.18081.13
Model name:         
Kiuwan model:       
Include patterns:   **/*
Exclude patterns:   **/*.min.js,**/*.Designer.vb,**/*Reference.vb,**/*Service.vb,**/*Silverlight.vb,**/*.designer.vb,**/*reference.vb,**/*service.vb,**/*silverlight.vb,**/*.Designer.cs,**/*Reference.cs,**/*Service.cs,**/*Silverlight.cs,**/*.designer.cs,**/*reference.cs,**/*service.cs,**/*silverlight.cs,**/.*,**/Pods/BuildHeaders/**/*.h,**/Pods/Headers/**/*.h
Create if needed:   true
Wait for results:   true
Encoding:           UTF-8
Ignore steps:       architecture,insights
Update engine: STARTED
Connecting to (34.252.119.191) -> api.kiuwan.com
Current Kiuwan Local Analyzer engine version: [No engine found]
Target Kiuwan Local Analyzer engine version: master.p492.q8302b.a1756.i222
Current Kiuwan Local Analyzer engine version: [No engine found]
Target Kiuwan Local Analyzer engine version: master.p492.q8302b.a1756.i222
Downloading Kiuwan Local Analyzer engine target version...
1913 KB downloaded
... Squashed ...
99833 KB downloaded
97 MB downloaded. Download complete!
Ready to update Kiuwan Local Analyzer engine...
Kiuwan Local Analyzer engine update done
Update engine: FINISHED
Analysis authorization: STARTED
Connecting to (34.252.119.191) -> api.kiuwan.com
Connecting to (34.252.119.191) -> api.kiuwan.com
ERROR running target "analysis.authorization" - Unexpected response status = 403; Reason: Access denied.
Kiuwan Local Analyzer found a problem in the current analysis:
Error returned from Kiuwan: Unexpected response status = 403; Reason: Access denied.
This is not a fatal error and you should be able to complete your analysis by following the steps shown in the Kiuwan Local Analyzer troubleshooting guide:
https://www.kiuwan.com/docs/display/K5/Troubleshooting
Exit code = 22
ERROR running target "run" - Unexpected response status = 403; Reason: Access denied.
KLA returned 22 Analysis finished successfully!
##[error]Kiuwan analysis failed! See messages above.
##[section]Finishing: Kiuwan baseline analysis

VSTS Agent log This is a snippet from the VSTS agent log regarding the build with the Kiuwan task.

[INFO Terminal] WRITE LINE: 2018-03-22 16:15:09Z: Running job: Build
[INFO JobDispatcher] Start renew job request 633 for job 8c4ef15e-2f21-47ea-a06f-dbe9f52191ef.
[INFO JobDispatcher] Successfully renew job request 633, job is valid till 3/22/2018 4:20:09 PM
[INFO ProcessInvoker] Starting process:
[INFO ProcessInvoker]   File name: 'C:\TfsData\Agent\bin\Agent.Worker.exe'
[INFO ProcessInvoker]   Arguments: 'spawnclient 2028 1916'
[INFO ProcessInvoker]   Working directory: 'C:\TfsData\Agent\bin'
[INFO ProcessInvoker]   Require exit code zero: 'False'
[INFO ProcessInvoker]   Encoding web name:  ; code page: ''
[INFO ProcessInvoker]   Force kill process on cancellation: 'True'
[INFO ProcessInvoker] Process started with process id 1792, waiting for process exit.
[INFO JobDispatcher] Send job request message to worker for job 8c4ef15e-2f21-47ea-a06f-dbe9f52191ef.
[INFO JobNotification] Entering JobStarted Notification
[INFO JobDispatcher] Successfully renew job request 633, job is valid till 3/22/2018 4:21:09 PM
[INFO JobDispatcher] Successfully renew job request 633, job is valid till 3/22/2018 4:22:09 PM
[INFO ProcessInvoker] Finished process with exit code 102, and elapsed time 00:02:25.6184062.
[INFO JobDispatcher] Worker finished for job 8c4ef15e-2f21-47ea-a06f-dbe9f52191ef. Code: 102
[INFO JobDispatcher] finish job request for job 8c4ef15e-2f21-47ea-a06f-dbe9f52191ef with result: Failed
[INFO Terminal] WRITE LINE: 2018-03-22 16:17:35Z: Job Build completed with result: Failed
[INFO JobDispatcher] Stop renew job request for job 8c4ef15e-2f21-47ea-a06f-dbe9f52191ef.
[INFO JobDispatcher] job renew has been canceled, stop renew job request 633.
[INFO JobNotification] Entering JobCompleted Notification
[ERR  MessageListener] Catch exception during get next message.
[ERR  MessageListener] System.Security.Cryptography.CryptographicException: RSA key file C:\TfsData\Agent\.credentials_rsaparams was not found
   at Microsoft.VisualStudio.Services.Agent.Listener.Configuration.RSAEncryptedFileKeyManager.GetKey()
   at Microsoft.VisualStudio.Services.Agent.Listener.MessageListener.GetMessageDecryptor(Aes aes, TaskAgentMessage message)
   at Microsoft.VisualStudio.Services.Agent.Listener.MessageListener.DecryptMessage(TaskAgentMessage message)
   at Microsoft.VisualStudio.Services.Agent.Listener.MessageListener.<GetNextMessageAsync>d__13.MoveNext()
[INFO MessageListener] Retriable exception: RSA key file C:\TfsData\Agent\.credentials_rsaparams was not found
[ERR  Terminal] WRITE ERROR: 2018-03-22 16:19:52Z: Agent connect error: RSA key file C:\TfsData\Agent\.credentials_rsaparams was not found. Retrying every 15 seconds until reconnected.
[INFO MessageListener] Sleeping for 15 seconds before retrying.
[INFO MessageListener] Message '344' received from session 'f39086a8-8752-44a9-a536-73f4369f05fd'.
[INFO JobDispatcher] Shutting down JobDispather. Make sure all WorkerDispatcher has finished.
[INFO JobDispatcher] Ensure WorkerDispather for job 8c4ef15e-2f21-47ea-a06f-dbe9f52191ef run to finish, cancel any running job.
[INFO JobDispatcher] Job request 8c4ef15e-2f21-47ea-a06f-dbe9f52191ef processed succeed.
[ERR  Terminal] WRITE ERROR: An error occurred: Unexpected character encountered while parsing value: b. Path '', line 0, position 0.
[ERR  AgentProcess] Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: b. Path '', line 0, position 0.
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at Microsoft.TeamFoundation.DistributedTask.WebApi.JsonUtility.FromString[T](String toDeserialize)
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.<RunAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.<RunAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.<RunAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Listener.Agent.<ExecuteCommand>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.VisualStudio.Services.Agent.Listener.Program.<MainAsync>d__1.MoveNext()
jsalado commented 6 years ago

Hello @tasadar2

Sorry for the inconvenience this is causing you.

The intention of the "Move-Item" is to move the installation of the Kiuwan Local Analyzer (KLA) to the agent root directory so it can be used in subsequent builds by that agent avoiding the overhead of downloading and installing the KLA every time. This command: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -command "Move-Item -Path 'C:\TfsData\Agent\12a2e36b-7948-4c20-b3d8-12754c3511d1\KiuwanLocalAnalyzer' -Destination 'C:\TfsData\Agent' should just move the KiuwanLocalAnalyzer directory in the 12a2e36b-7948-4c20-b3d8-12754c3511d1 directory one directory up. The result should be a new directory C:\TfsData\Agent\KiuwanLocalAnalyzer it shouldn't remove any files in the C:\TfsData\Agent directory. The task doesn't do any clean up

I'm investigating the issue. I have already tried with my local TFS and an agent I have locally and couldn't reproduced it. Please share with me some more info about your setup:

In the case that you have control over the agent I can propose a work around until we find out where the problem is:

  1. Download the KLA from your Kiuwan account.
  2. Unzip the downloaded zip file anywhere you want in the TFS agent machine
  3. Define the KIUWAN_HOME variable in your build definition pointing to where you unzip the KLA (e.g. KIUWAN_HOME=C:\\KiuwanLocalAnalyzer).
  4. Run your task again.

This will avoid the task downloading and installing the KLA so it doesn't need to do the Move-Item.

tasadar2 commented 6 years ago

If there is no cleanup in the task, then I have to assume that TFS is responsible. I wonder if the specified toolPath for the Kiuwan Analyzer is being cleaned.

Troubleshooting After looking through the code a bit, it looks like the environment variable Agent.TempDirectory is being overridden to the Agent.HomeDirectory. My bet is that TFS cleans the Agent.TempDirectory during the cleanup phase at the end of the build. And if this set to the Agent.HomeDirectory, then it will be cleaned instead. Although the released version is newer than the source: https://github.com/kiuwan/tfs-vsts-extension/blob/master/baseline-analysis-task/index.ts#L242

I tried running a build with a sleep after the Kiuwan task, and sure enough, the directory structure is intact. It is only after the cleanup at the end of the build, that the directory/agent is removed.

Instead of overriding the Agent.TempDirectory, I would suggest letting it download to the default temp location, caching the extracted directory, which copies the directory to the Agent/_work/_tool directory. This is then registerred as a tool to be used later.

// Register tool
extPath = await ttl.cacheDir(extPath, 'KiuwanLocalAnalyzer', '2.1.2');

// Lookup tool
let klaInstallPath = ttl.findLocalTool('KiuwanLocalAnalyzer', '2.1.2');
if (!klaInstallPath) {
    klaInstallPath = downloadInstallKla(...)
}

Although the source seems older than the task's code, https://github.com/kiuwan/tfs-vsts-extension/pull/3

Environment This is a local instance of TFS with as local agent. We are testing it locally before installing it in our on-premise TFS Test environment.

kiuwan commented 6 years ago

Hi @tasadar2,

That seems to be the issue and your pull request should fix it.

We'll commit and merge your pull request. But you are right, the code here is older than the currently published extension. There is a local branch pending to be pushed. We'll do that too so the master is current, making sure the the override of the Agent temp dir is gone from that branch and after running some tests.

Thank you.

kiuwan commented 6 years ago

Hi @tasadar2,

just committed a new version that extends oyur fix for the override of the temp directory. It brings the code up to date with the publish version of the extension in the marketplace. And adds some refactoring to avoid duplication of code.

If you test the new version of the extension in your local TFS please, let us know if it works as expected now so we can close the issue.

Thanks.

tasadar2 commented 6 years ago

Looks like that fixed it.