search

xamarin / xamarin-macios

.NET for iOS, Mac Catalyst, macOS, and tvOS provide open-source bindings of the Apple SDKs for use with .NET managed languages such as C#
Other
2.42k stars 504 forks source link

[CI] Split tests to a post-build pipeline. #20534

Closed mandel-macaque closed 2 months ago

mandel-macaque commented 2 months ago

1Es is problematic and has a lot of issues on how we use the matrix strategies. We have two options, either wait for 1ES to implement the needed changes or move all the tests out to two different pipelines that won't need to be 1ES complient.

In the long run moving out of the build piepline is going to give us more flexibility and will move the project away from depending on a thridparty to fix the yaml issues.

Even when landing this PR we will not have the tests removed, we will do that after we have been able to test that things work as expected.

vs-mobiletools-engineering-service2 commented 2 months ago

:books: [PR Build] Artifacts :books:

Packages generated

View packages * [xamarin.ios-17.3.0.377.pkg](https://dl.internalx.com/wrench/merge/ff0d302c36a6cca29c20846ff2d4be696dfcebbc/9498530/package/xamarin.ios-17.3.0.377.pkg) * [xamarin.mac-9.3.0.377.pkg](https://dl.internalx.com/wrench/merge/ff0d302c36a6cca29c20846ff2d4be696dfcebbc/9498530/package/xamarin.mac-9.3.0.377.pkg)

Pipeline on Agent Hash: [PR build]

vs-mobiletools-engineering-service2 commented 2 months ago

:computer: [PR Build] Tests on macOS X64 - Mac Sonoma (14) passed :computer:

:white_check_mark: All tests on macOS X64 - Mac Sonoma (14) passed.

Pipeline on Agent Hash: [PR build]

vs-mobiletools-engineering-service2 commented 2 months ago

:computer: [PR Build] Tests on macOS M1 - Mac Ventura (13) passed :computer:

:white_check_mark: All tests on macOS M1 - Mac Ventura (13) passed.

Pipeline on Agent Hash: [PR build]

vs-mobiletools-engineering-service2 commented 2 months ago

:computer: [PR Build] Tests on macOS M1 - Mac Monterey (12) passed :computer:

:white_check_mark: All tests on macOS M1 - Mac Monterey (12) passed.

Pipeline on Agent Hash: [PR build]

vs-mobiletools-engineering-service2 commented 2 months ago

:computer: [PR Build] Tests on macOS M1 - Mac Big Sur (11) passed :computer:

:white_check_mark: All tests on macOS M1 - Mac Big Sur (11) passed.

Pipeline on Agent Hash: [PR build]

vs-mobiletools-engineering-service2 commented 2 months ago

:computer: [CI Build] Windows Integration Tests passed :computer:

:white_check_mark: All Windows Integration Tests passed.

Pipeline on Agent Hash: d7a2f3524859c332a46682bb62b624aef21860bb [PR build]

vs-mobiletools-engineering-service2 commented 2 months ago

:white_check_mark: API diff for current PR / commit

Legacy Xamarin (No breaking changes) * ~iOS~ (no change detected) * ~tvOS~ (no change detected) * ~watchOS~ (no change detected) * ~macOS~ (no change detected)
NET (empty diffs) * ~iOS~: (empty diff detected) * ~tvOS~: (empty diff detected) * ~MacCatalyst~: (empty diff detected) * ~macOS~: (empty diff detected)

:white_check_mark: API diff vs stable

Legacy Xamarin (No breaking changes) * iOS: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/ios-api-diff.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/5a9d1ec2e0fddc1ac02b69bc8db75cd1) (No breaking changes) * tvOS: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/tvos-api-diff.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/ed3450aca8f04bde0a8e300fa5bf2850) (No breaking changes) * watchOS: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/watchos-api-diff.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/6c70fe2b09f47c6fe0157b4494ea8337) (No breaking changes) * macOS: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/mac-api-diff.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/2c8e745a72736c5813b0aff205ebb1b9) (No breaking changes)
.NET (No breaking changes) * iOS: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/diff/dotnet/Microsoft.iOS.Ref/ref/net8.0/Microsoft.iOS.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/55bd1deb6d419f74461ef8a020fa26fb) (No breaking changes) * tvOS: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/diff/dotnet/Microsoft.tvOS.Ref/ref/net8.0/Microsoft.tvOS.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/dddbb13a8885532da395fcbfeb502886) (No breaking changes) * MacCatalyst: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/diff/dotnet/Microsoft.MacCatalyst.Ref/ref/net8.0/Microsoft.MacCatalyst.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/78f5a6f134dcd2aae14cdadcec5f7294) (No breaking changes) * macOS: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/diff/dotnet/Microsoft.macOS.Ref/ref/net8.0/Microsoft.macOS.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/b885541c0b1c771c8094ea44f7bbe8c9) (No breaking changes) * Microsoft.iOS vs Microsoft.MacCatalyst: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/diff/dotnet/iOS-MacCatalyst-diff/Microsoft.iOS.Ref/ref/net8.0/Microsoft.iOS.MacCatalyst.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/d04f6f653873b94b81d073ec2e8b39f0)
Legacy Xamarin (stable) vs .NET * iOS: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/diff/dotnet/legacy-diff/Microsoft.iOS.Ref/ref/net8.0/Microsoft.iOS.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/c521d2823ca2430cf685b0670ee4f83f) * tvOS: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/diff/dotnet/legacy-diff/Microsoft.tvOS.Ref/ref/net8.0/Microsoft.tvOS.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/e3b4e80fe6dfb75e918372dee5d3c26c) * macOS: [vsdrops](https://vsdrop.corp.microsoft.com/file/v1/xamarin-macios/detected-changes/20240429.15/9498530-1/;/stable-api-comparison/diff/dotnet/legacy-diff/Microsoft.macOS.Ref/ref/net8.0/Microsoft.macOS.html) [gist](https://gist.github.com/vs-mobiletools-engineering-service2/f4aaa81d832e7ac9d0cc3fc63463bfa7)

:information_source: Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent Hash: d7a2f3524859c332a46682bb62b624aef21860bb [PR build]

vs-mobiletools-engineering-service2 commented 2 months ago

:fire: [CI Build] Test results :fire:

Test results

:x: Tests failed on VSTS: test results

0 tests crashed, 3 tests failed, 167 tests passed.

Failures

:x: dotnettests tests (tvOS)

1 tests failed, 0 tests passed.
* DotNet tests: Failed (Execution failed with exit code 1)

Html Report (VSDrops) Download

:x: linker tests

1 tests failed, 64 tests passed.
* trimmode copy/tvOS - simulator/Debug [dotnet]: Crashed

Html Report (VSDrops) Download

:x: monotouch tests (MacCatalyst)

1 tests failed, 6 tests passed.
* monotouch-test/Mac Catalyst [dotnet]/Release (all optimizations) [dotnet]: Failed (Test run failed. Tests run: 2978 Passed: 2844 Inconclusive: 10 Failed: 1 Ignored: 133)

Html Report (VSDrops) Download

Successes

:white_check_mark: cecil: All 1 tests passed. Html Report (VSDrops) Download :white_check_mark: dotnettests (iOS): All 1 tests passed. Html Report (VSDrops) Download :white_check_mark: dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download :white_check_mark: dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download :white_check_mark: dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download :white_check_mark: framework: All 8 tests passed. Html Report (VSDrops) Download :white_check_mark: fsharp: All 7 tests passed. Html Report (VSDrops) Download :white_check_mark: generator: All 2 tests passed. Html Report (VSDrops) Download :white_check_mark: install-source: All 1 tests passed. Html Report (VSDrops) Download :white_check_mark: interdependent-binding-projects: All 7 tests passed. Html Report (VSDrops) Download :white_check_mark: introspection: All 8 tests passed. Html Report (VSDrops) Download :white_check_mark: mac-binding-project: All 1 tests passed. Html Report (VSDrops) Download :white_check_mark: mmp: All 2 tests passed. Html Report (VSDrops) Download :white_check_mark: mononative: All 6 tests passed. Html Report (VSDrops) Download :white_check_mark: monotouch (iOS): All 11 tests passed. Html Report (VSDrops) Download :white_check_mark: monotouch (macOS): All 8 tests passed. Html Report (VSDrops) Download :white_check_mark: monotouch (tvOS): All 11 tests passed. Html Report (VSDrops) Download :white_check_mark: monotouch (watchOS): All 4 tests passed. Html Report (VSDrops) Download :white_check_mark: msbuild: All 2 tests passed. Html Report (VSDrops) Download :white_check_mark: mtouch: All 1 tests passed. Html Report (VSDrops) Download :white_check_mark: xammac: All 3 tests passed. Html Report (VSDrops) Download :white_check_mark: xcframework: All 8 tests passed. Html Report (VSDrops) Download :white_check_mark: xtro: All 2 tests passed. Html Report (VSDrops) Download

Pipeline on Agent Hash: [PR build]

mandel-macaque commented 2 months ago

We are obj with he failures, we wanted to make sure the tests would run and the result would be posted.