[tests] Apple mobile test coverage improvements

[kotlarmilos]

Description

This issue lists improvements in Apple mobile infrastructure and test coverage. The tasks are categorized into four main areas: test coverage, reliability, scalability, and monitoring.

Expand runtime tests coverage

The primary goal is to align test coverage with the production environments, ensuring validation across all scenarios. Currently, only legacy runtime tests are executed on Apple mobile platforms using the interpreter, with a subset of smoke tests executed with Native AOT. The library and functional tests utilize the Mono AOT compiler. List of tasks:

• [ ] Document and implement test environment that reflects production usage for runtime, library, and functional tests
• [ ] Enable building runtime, library, and functional tests locally with a single command
• [ ] Improve Mono test coverage
  • [ ] https://github.com/dotnet/runtime/pull/92523
  • [ ] Implement Xunit runtime tests coverage using Mono runtime
• [ ] Improve Native AOT test coverage
  • [ ] https://github.com/dotnet/runtime/issues/81075
  • [ ] https://github.com/dotnet/runtime/issues/91871
  • [ ] https://github.com/dotnet/runtime/issues/81075
  • [ ] https://github.com/dotnet/runtime/issues/93447
  • [x] https://github.com/dotnet/runtime/issues/89437
  • [ ] Investigate how we can configure NativeAOT runtime tests to include Microsoft.NET.Sdk.props and with it Microsoft.NETCoreSdk.BundledVersions.props which defines NETCoreSdkPortableRuntimeIdentifier
• [ ] https://github.com/dotnet/runtime/issues/92129

Scalability

The primary goal is to improve the overall test build and execution speed. List of tasks:

• [ ] https://github.com/dotnet/runtime/issues/95060
• [x] https://github.com/dotnet/runtime/pull/100669
• [ ] Enable trimming for runtime tests
• [ ] https://github.com/dotnet/runtime/issues/89917
• [ ] Reduce test coverage redundancy between Apple mobile jobs and other non-mobile jobs
• [x] https://github.com/dotnet/runtime/issues/98213

Reliability

The primary goal is to establish a stable infrastructure environment for test execution. List of tasks:

• [x] https://github.com/dotnet/runtime/issues/95862
• [ ] https://github.com/dotnet/runtime/issues/87187
• [ ] https://github.com/dotnet/runtime/issues/82637
• [ ] Rename all the .props, .targets explicitly related to compilation on Helix - see comment
• [ ] Improve error and stacktrace logging
• [ ] Builds on Helix should not reuse publish folder for different builds tasks - see comment

Monitoring

• [ ] Add llvm check similar to this https://github.com/dotnet/runtime/pull/92644/files#diff-058f1e9bc8e4b23fb2d85884c92317671d13ecccb1cafa95f0f6c6430a600423 to the fullAOT jobs

The primary goal of monitoring should be enabling effective triage of active issues and tracking the reliability of tests and infrastructure over time. A list of tasks will be added.

[ghost]

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger See info in area-owners.md if you want to be subscribed.

Issue Details

---

## Description This issue aims to expand the coverage of iOS runtime tests and improve the build process. ### Expand runtime tests coverage on iOS Currently, only `tracing/eventpipe` is running on iOS/iOSSimulator using interpreter as a fallback. This issue aims to expand the coverage of iOS runtime tests by including other testing groups and enabling full AOT compilation. ### Parallel AOT compilation To improve build time, the AOT compilation could be moved to Helix and executed simultaneously before the test execution. Additionally, unifying AOT compilation in one target would improve code maintenance as several targets use `MonoAOTCompiler` and `AppleAppBuilderTask` tasks. ### Split build and test execution To mitigate various transient issues, build and test execution could be divided into separate steps. The test build should build the repository, create tests, and zip artifacts (as done in `build-test-job.yml`). The test execution should download the artifacts, unzip them, and send the tests to Helix for AOT compilation and execution (as done in `run-test-job.yml`). ## Tasks - [ ] Identify other testing groups that could be executed on iOS/iOSSimulator and enable full AOT compilation for iOS/iOSSimulator tests - [ ] `baseservices` - [ ] `JIT` - [ ] `GC` - [ ] `ilverify` - [ ] `Interop` - [ ] `Loader` - [ ] `tracing` - [ ] Move AOT compilation to Helix and execute it in parallel with test execution - [ ] Unify AOT compilation to improve code maintenance - [ ] Separate build and test execution into separate builds

Author:	kotlarmilos
Assignees:	kotlarmilos
Labels:	`test-enhancement`, `area-Infrastructure-mono`, `os-ios`
Milestone:	Future

[kotlarmilos]

To mitigate various transient issues, build and test execution could be divided into separate steps. The test build should build the runtime and tests, and zip artifacts. The test execution should download the artifacts, unzip them, and send the tests to Helix for AOT compilation and execution. Additionally, a single build could be shared across runtime, library, and functional tests.

@steveisok I think you mentioned once this may not be a good approach. My assumption is that it may reduce costs and development time in cases when transient issues occur. What could be the potential downsides of such an approach? Are there some bottlenecks?