[Android][infra] Increase the Android test coverage across different OS versions

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.

https://docs.microsoft.com/dotnet/core/

MIT License

14.56k stars 4.54k forks source link

[Android][infra] Increase the Android test coverage across different OS versions #102555

Open matouskozak opened 1 month ago

matouskozak commented 1 month ago

For .NET 9, we support Android API version of 21 and newer (https://github.com/dotnet/core/blob/main/release-notes/9.0/supported-os.md#android). Our CI infrastructure only test a fixed API version: https://github.com/dotnet/runtime/blob/9a18cf5d88f1a43097646f147140a7078760ec4e/eng/pipelines/libraries/helix-queues-setup.yml#L94-L98

We should explore the possibility of having coverage over both older and newer Android API version for the Libraries tests.

TODO:

Identify which Libraries tests should be evaluated on different API versions (prone to breaking Android API changes)
Estimate cost of new CI infrastructure jobs for Android testing
Introduce new Android jobs covering older/newer OS versions

dotnet-policy-service[bot] commented 1 month ago

Tagging subscribers to 'arch-android': @steveisok, @akoeplinger See info in area-owners.md if you want to be subscribed.

dotnet-policy-service[bot] commented 1 month ago

Tagging subscribers to this area: @directhex See info in area-owners.md if you want to be subscribed.

matouskozak commented 1 month ago

fyi: @vitek-karas @simonrozsival

akoeplinger commented 1 month ago

We do have Helix queues for the newer API levels, it just never was worth the CI cost and pain of triaging flaky issues to keep them enabled all the time. We used a temporary PR (https://github.com/dotnet/runtime/pull/64341) to trigger the "full" set from time to time, but nowadays having a separate pipeline probably makes more sense.

matouskozak commented 1 month ago

We do have Helix queues for the newer API levels, it just never was worth the CI cost and pain of triaging flaky issues to keep them enabled all the time. We used a temporary PR (#64341) to trigger the "full" set from time to time, but nowadays having a separate pipeline probably makes more sense.

I think a temporary PR might be a good idea as well. Our initial idea was to do ~weekly runs but as you mentioned it depends on the CI cost and flakiness of the issues. I think it also depends on how many really issues we can discover using this approach and perhaps just a selected set of Libraries test could be run if they are prone to breaking changes by Android OS bumps.

akoeplinger commented 1 month ago

perhaps just a selected set of Libraries test could be run if they are prone to breaking changes by Android OS bumps.

IMO trying to guess which tests could be broken is kinda impossible, much easier to run everything. If you only run weekly it should be manageable to deal with the flaky issues.