search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.22k stars 3.83k forks source link

Investigate Batch test failures with new HttpRecorder #7512

Open cormacpayne opened 5 years ago

cormacpayne commented 5 years ago

With the latest version of HttpRecorder, there are two failing Batch tests:

Both tests succeed during re-record and when run live, but fail in playback locally and on Jenkins. This issue is to investigate the cause of these failures and provide a fix in whatever framework the issue was introduced into.

The current failure is the following:

Newtonsoft.Json.JsonReaderException: 'Error reading string. Unexpected token: StartArray. Path 'Content-Type'.'

The corresponding content data from this exception is the following:

 {
  "Transfer-Encoding": "chunked",
  "request-id": "c0d07c42-bbcb-45f9-bbf7-535a8d631460",
  "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
  "X-Content-Type-Options": "nosniff",
  "DataServiceVersion": "3.0",
  "ocp-creation-time": "Tue, 05 Jun 2018 17:11:00 GMT",
  "ocp-batch-file-isdirectory": "False",
  "ocp-batch-file-url": "https%3A%2F%2Fmatthchrwestcentralus.westcentralus.batch.azure.com%2Fpools%2FtestPool%2Fnodes%2Ftvm-892191217_1-20180605t170754z%2Ffiles%2Fstartup%2Fstdout.txt",
  "Date": "Tue, 05 Jun 2018 23:37:06 GMT",
  "Server": "Microsoft-HTTPAPI/2.0",
  "azSdkTestPlayBackMode": "true",
  "Content-Type": [
    "text/plain; charset=utf-8",
    "text/plain"
  ],
  "Last-Modified": "Tue, 05 Jun 2018 17:11:00 GMT",
  "Content-Length": "7"
}

There appears to be an issue with the Content-Type array having two values rather than just one text/plain value.

matthchr commented 4 years ago

Reactivating this - these tests still pass in Record mode but fail in Playback. Interestingly the error is different than Cormac shared above.

Test Name:  Microsoft.Azure.Commands.Batch.Test.ScenarioTests.FileTests.TestGetNodeFileContentByComputeNode
Test Outcome:   Failed
Result Message: System.Management.Automation.ActionPreferenceStopException : The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: Queue empty for request HEAD /pools/testPool/nodes/tvmps_6aa5db52f9a032cf74ccd9dd8efc4deabc213c1b27fb36cd49b2bad8bc2ab3b9_d/files/startup%5Cstdout.txt?api-version=2019-08-01.10.0
Result StandardOutput:  
Test-GetNodeFileContentByComputeNode 'testPool' 'tvmps_6aa5db52f9a032cf74ccd9dd8efc4deabc213c1b27fb36cd49b2bad8bc2ab3b9_d' 'startup\stdout.txt' 'hello'
Caught Exception: System.Management.Automation.ActionPreferenceStopException: The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: Queue empty for request HEAD /pools/testPool/nodes/tvmps_6aa5db52f9a032cf74ccd9dd8efc4deabc213c1b27fb36cd49b2bad8bc2ab3b9_d/files/startup%5Cstdout.txt?api-version=2019-08-01.10.0
   at System.Management.Automation.Runspaces.PipelineBase.Invoke(IEnumerable input)
   at System.Management.Automation.PowerShell.Worker.ConstructPipelineAndDoWork(Runspace rs, Boolean performSyncInvoke)
   at System.Management.Automation.PowerShell.Worker.CreateRunspaceIfNeededAndDoWork(Runspace rsToUse, Boolean isSync)
   at System.Management.Automation.PowerShell.CoreInvokeHelper[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.CoreInvoke[TInput,TOutput](PSDataCollection`1 input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.CoreInvoke[TOutput](IEnumerable input, PSDataCollection`1 output, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.Invoke(IEnumerable input, PSInvocationSettings settings)
   at System.Management.Automation.PowerShell.Invoke()
   at Microsoft.WindowsAzure.Commands.ScenarioTest.EnvironmentSetupHelper.RunPowerShellTest(String[] scripts) in C:\work\github\azure-powershell\tools\ScenarioTest.ResourceManager\EnvironmentSetupHelper.cs:line 571
Message: The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: Queue empty for request HEAD /pools/testPool/nodes/tvmps_6aa5db52f9a032cf74ccd9dd8efc4deabc213c1b27fb36cd49b2bad8bc2ab3b9_d/files/startup%5Cstdout.txt?api-version=2019-08-01.10.0
PowerShell Error Record: Queue empty for request HEAD /pools/testPool/nodes/tvmps_6aa5db52f9a032cf74ccd9dd8efc4deabc213c1b27fb36cd49b2bad8bc2ab3b9_d/files/startup%5Cstdout.txt?api-version=2019-08-01.10.0
Exception:System.InvalidOperationException: Queue empty for request HEAD /pools/testPool/nodes/tvmps_6aa5db52f9a032cf74ccd9dd8efc4deabc213c1b27fb36cd49b2bad8bc2ab3b9_d/files/startup%5Cstdout.txt?api-version=2019-08-01.10.0
   at Microsoft.Azure.Test.HttpRecorder.HttpMockServer.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at Microsoft.Azure.Batch.Protocol.FileOperations.GetPropertiesFromComputeNodeWithHttpMessagesAsync(String poolId, String nodeId, String filePath, FileGetPropertiesFromComputeNodeOptions fileGetPropertiesFromComputeNodeOptions, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Batch.Protocol.BatchRequestBase`2.ExecuteRequestWithCancellationAsync(Func`2 func)
   at Microsoft.Azure.Batch.Protocol.BatchRequestBase`2.ExecuteRequestAsync()
   at Microsoft.Azure.Batch.Protocol.BatchRequestBase`2.ExecuteRequestAsync()
   at Microsoft.Azure.Batch.ProtocolLayer.ProcessAndExecuteBatchRequest[TResponse](IBatchRequest`1 request, BehaviorManager bhMgr)
   at Microsoft.Azure.Batch.ProtocolLayer.GetNodeFilePropertiesByNode(String poolId, String computeNodeId, String filePath, BehaviorManager bhMgr, CancellationToken cancellationToken)
   at Microsoft.Azure.Batch.PoolOperations.GetNodeFileAsyncImpl(String poolId, String computeNodeId, String filePath, BehaviorManager bhMgr, CancellationToken cancellationToken)
   at Microsoft.Azure.Batch.UtilitiesInternal.WaitAndUnaggregateException[T](Task`1 task)
   at Microsoft.Azure.Batch.UtilitiesInternal.WaitAndUnaggregateException[T](Task`1 task, IEnumerable`1 rootBehaviors, IEnumerable`1 additionalBehaviors)
   at Microsoft.Azure.Batch.PoolOperations.GetNodeFile(String poolId, String computeNodeId, String filePath, IEnumerable`1 additionalBehaviors)
   at Microsoft.Azure.Commands.Batch.Models.BatchClient.ListNodeFilesByComputeNode(ListNodeFileOptions options) in C:\work\github\azure-powershell\src\Batch\Batch\Models\BatchClient.Files.cs:line 111
   at Microsoft.Azure.Commands.Batch.Models.BatchClient.ListNodeFiles(ListNodeFileOptions options) in C:\work\github\azure-powershell\src\Batch\Batch\Models\BatchClient.Files.cs:line 49
   at Microsoft.Azure.Commands.Batch.GetBatchNodeFileCommand.ExecuteCmdletImpl() in C:\work\github\azure-powershell\src\Batch\Batch\Files\GetBatchNodeFileCommand.cs:line 112
   at Microsoft.Azure.Commands.Batch.BatchCmdletBase.ExecuteCmdlet() in C:\work\github\azure-powershell\src\Batch\Batch\BatchCmdletBase.cs:line 55
   at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Details:
Script Stack Trace: at Test-GetNodeFileContentByComputeNode, C:\work\github\azure-powershell\src\Batch\Batch.Test\bin\Debug\netcoreapp2.2\ScenarioTests\FileTests.ps1: line 61
at <ScriptBlock>, <No file>: line 1
: Target:

@markcowl - it would be nice to get this looked at and at least understand if this is some expected limitation of the recording framework, or a bug.