search

dotnet / performance

This repo contains benchmarks used for testing the performance of all .NET Runtimes
MIT License
700 stars 273 forks source link

[maui-scenarios] MAUI scenarios and XHarness versions #4574

Open ivanpovazan opened 2 weeks ago

ivanpovazan commented 2 weeks ago

Description

Problem 1: using an outdated version

MAUI scenarios on Android are using an outdated version of XHarness. More specifically, the version used in perf jobs is: https://github.com/dotnet/performance/blob/acac700c6f839490f3535297363f8fc658b269a3/eng/performance/maui_scenarios_android.proj#L7

For reference, the current version of XHarness is: 10.0.0-prerelease.24524.9

Problem 2: using outdated commands

Bumping the version manually will not be the only fix/change for this issue as the code in: https://github.com/dotnet/performance/blob/acac700c6f839490f3535297363f8fc658b269a3/src/scenarios/shared/androidhelper.py#L25 is invoking xharness android state --adb which is not a supported command anymore

Problem 3: mismatch between referenced xharness versions

MAUI scenarios for iOS are using a different xharness version: https://github.com/dotnet/performance/blob/acac700c6f839490f3535297363f8fc658b269a3/eng/performance/maui_scenarios_ios.proj#L6

It is recommended to align the xharness references.

NOTE

An additional consideration would be to switch from hardcoding xharness versions to use darc subscriptions instead.

Security

While this currently "works" on CI, any changes to the tools or its references (like adb) will not be available for perf testing. With that, all recent improvements regarding security/SDL work on XHarness will not be included.

/cc: @vitek-karas

ivanpovazan commented 3 days ago

@LoopedBard3 please feel free to link the testing CI runs here so we can help if needed investigating the failures.

matouskozak commented 2 days ago

Based on the logs from https://dev.azure.com/dnceng/internal/_build/results?buildId=2590246&view=logs&j=efa3ffcd-91e9-5b69-9db7-650958b3131c&t=a635f724-5afe-5774-89bd-de12fd2d4e6e:

(.venv) D:\h\w\A39508DB\w\B8830A85\e>python test.py devicestartup --device-type android --package-path pub\com.companyname.netandroiddefault-Signed.apk --package-name com.companyname.NetAndroidDefault --scenario-name "Device Startup - .NET Android Default"  --upload-to-perflab-container 
[2024/11/26 16:51:02][INFO] ----------------------------------------------
[2024/11/26 16:51:02][INFO] Initializing logger 2024-11-26 16:51:02.220086
[2024/11/26 16:51:02][INFO] ----------------------------------------------
[2024/11/26 16:51:02][INFO] Clearing potential previous run nettraces
[2024/11/26 16:51:02][INFO] Preparing ADB
[2024/11/26 16:51:02][INFO] $ dotnet exec D:\h\w\A39508DB\p\microsoft.dotnet.xharness.cli\10.0.0-prerelease.24524.9\tools\net8.0\any\Microsoft.DotNet.XHarness.CLI.dll android adb -- shell wm size
[2024/11/26 16:51:04][INFO] Physical size: 1080x2340
[2024/11/26 16:51:04][INFO] * daemon not running; starting now at tcp:5037
[2024/11/26 16:51:04][INFO] * daemon started successfully
['Device Startup - .NET Android Default' END OF WORK ITEM LOG: Command timed out, and was killed]

it appears that we are executing XHarness and getting the output successfully.

However, I did some experiments in my own branch and it appears that the XHarness command is executed but gets stuck inside https://github.com/dotnet/performance/blob/9d4bbf4f00fa8392010e5cb3aa95bbe81ae9fd03/scripts/performance/common.py#L274-L302

My guess is that something changed in XHarness + adb which is preventing Python Popen to successfully finish (XHarness or subprocess adb not exiting?). I couldn't reproduce this locally so I think we might need to take one of the machines out of the Helix queue and investigate it there.

For reference, https://github.com/dotnet/xharness/pull/782 PR that added xharness android adb -- functionality.

Update

Reproduction tries with xharness 10.0.0-prerelease.24524.9: