search

dotnet / dnceng

.NET Engineering Services
MIT License
25 stars 18 forks source link

[android] Failing E2E Android - Devices Helix Tests Build_Debug tests #3656

Open ivanpovazan opened 2 months ago

ivanpovazan commented 2 months ago

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=750141

Build leg reported

E2E Android - Devices Helix Tests Build_Debug failed

Pull Request

https://github.com/dotnet/xharness/pull/1233

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

{
  "ErrorMessage": "crit: Install failure: Test command cannot continue",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

@dotnet/dnceng

Release Note Category

Additional information about the issue reported

No response

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=750141 Error message validated: [crit: Install failure: Test command cannot continue] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 7/29/2024 6:26:51 PM UTC

Report

Build Definition Test Pull Request
825305 dotnet/runtime System.Threading.ThreadPool.Tests.WorkItemExecution
823678 dotnet/arcade System.Buffers.Tests-arm64-v8a.WorkItemExecution
823581 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
822315 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
822026 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1282
822072 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15087
819558 dotnet/runtime System.Security.Cryptography.OpenSsl.Tests.WorkItemExecution
819157 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1279
819143 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
819033 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1277
818848 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1277
818803 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1277
815614 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15087
817968 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1277
815834 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
815712 dotnet/xharness System.Buffers.Tests-x86.WorkItemExecution dotnet/xharness#1276
812840 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
810489 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
810474 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
809758 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1273
809552 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1272
809474 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1271
809392 dotnet/arcade System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/arcade#15087
809353 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1270
809244 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
809163 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1268
808489 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
808108 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1268
808047 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1267
807953 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1267
807915 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution
807376 dotnet/xharness System.Buffers.Tests-x86.WorkItemExecution
806990 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15020
806807 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1266
806688 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution
805484 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15071
805447 dotnet/arcade System.Buffers.Tests-arm64-v8a.WorkItemExecution
805368 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15072
804447 dotnet/arcade System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/arcade#15072
804445 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15071
805076 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15063
803820 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
803706 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1265
803714 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15063
803275 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1265
802212 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15063
802013 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
801990 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
801740 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15020
801705 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1264
801644 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1263
801602 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
801593 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1262
800079 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
800031 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1261
797829 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1260
797826 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
797553 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1259

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
1 11 58
ivanpovazan commented 1 week ago

Problem

This issue seems to be occurring more and more, and xharness CI is mostly red.

The problem comes from the fact that in some jobs we are testing debug apks on physical Android devices, which gives:

adb: failed to install D:\h\w\A8C809B3\w\A92C0917\e\System.Buffers.Tests-arm64-v8a.apk: Failure [INSTALL_FAILED_VERIFICATION_FAILURE: Package Verification Result]

[02:24:27] crit: Install failure: Test command cannot continue

(link to the latest failure on DNCENGWIN-007 : https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-xharness-refs-heads-release-90-68d2120ee0c9454f80/System.Buffers.Tests-arm64-v8a.Attempt.3/1/console.03621459.log?helixlogtype=result

Installing such apks can be rejected by the device, if Verify apps over USB setting is enabled on it.

Assumption

My assumption is that the tethered devices were initially configured to have this verification disabled in device settings - allowing debug apks to be installed and tested, and that a factory reset happened recently, which caused switching this setting on again and caused the failures we are now seeing.

Ask

DNCENGWIN-070 had a successful run https://dev.azure.com/dnceng-public/public/_build/results?buildId=811051&view=logs&j=00f02412-5f5e-5b12-0780-d2e57de065ba&t=b1200827-0fdc-54c0-1776-c7b8da453b26 while DNCENGWIN-007 seems to fail constantly. Checking the Verify apps over USB setting on tethered devices on these machines would help us figure out the next steps.

@ilyas1974, could someone check this out?

ilyas1974 commented 1 week ago

Hi Ivan, I have taken DNCENG-007 out of rotation so no additional jobs will use that system while we investigate the failure you mentioned above. I have created https://portal.microsofticm.com/imp/v5/incidents/details/546521338 to have the system in question looked at.

ivanpovazan commented 1 week ago

Thank you! Should I add some context on the created incident ticket?