[Breaking Change] Runtime exceptions aggregate

[konrad-jamrozik]

Note: the list below is competing for priority with

• https://github.com/Azure/azure-sdk-tools/issues/7795

This is an aggregate issue for Breaking Change runtime exceptions:

• [ ] https://github.com/Azure/azure-sdk-tools/issues/5981
• [ ] https://github.com/Azure/azure-sdk-tools/issues/6036
  • https://github.com/Azure/azure-sdk-tools/issues/6998
• [ ] https://github.com/Azure/azure-sdk-tools/issues/6168

And from the old, pre-transfer backlog:

• https://github.com/Azure/openapi-diff/issues/271
• https://github.com/Azure/openapi-diff/issues/258
• https://github.com/Azure/openapi-diff/issues/241
• https://github.com/Azure/openapi-diff/issues/239

Related issue:

• https://github.com/Azure/azure-sdk-tools/issues/6859

Errors reported after 9/11

I.e. after the runtime exception fix got merged and deployed.

• https://github.com/Azure/azure-sdk-tools/issues/6998
• https://github.com/Azure/azure-sdk-tools/issues/6999

References

For more, see comments below.

Konrad's private OneNote page for this issue is #6169 [Breaking Change] Runtime exceptions aggregate

[konrad-jamrozik]

Another case:

• https://github.com/Azure/azure-rest-api-specs/pull/23816#issuecomment-1535157360

[konrad-jamrozik]

Another runtime failure seen here:

• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2786392&view=logs&j=98f87804-8e1f-5655-af02-e80aefa7aa97&t=65457206-7ab6-5966-2a2f-9ae3efa14a03&l=192

Albeit this time the tool did generate some results:

• https://github.com/Azure/azure-rest-api-specs/pull/24059/checks?check_run_id=13597926900
• https://github.com/Azure/azure-rest-api-specs/pull/24059

[konrad-jamrozik]

Another occurrence, by Jessica Li on 6/29/2023:

• https://github.com/Azure/azure-rest-api-specs/pull/24387/checks?check_run_id=14608165051
  • Team discussion

[konrad-jamrozik]

Info from Ray about known limitation in this Teams discussion:

• https://github.com/Azure/azure-rest-api-specs-pr/pull/12695

There's a known limitation in the pipeline to remove the label when the new commit was pushed to the PR to remove the breaking change.

[konrad-jamrozik]

Another case:

• https://github.com/Azure/azure-sdk-tools/issues/6526

[konrad-jamrozik]

Another case:

• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2962440&view=logs&j=98f87804-8e1f-5655-af02-e80aefa7aa97&t=65457206-7ab6-5966-2a2f-9ae3efa14a03&l=52
• Context on Teams

Some clues from the error log:

ERROR: Schema violation: Missing required property: name\n - file:///mnt/vss/work/1/azure-rest-api-specs/specification/cognitiveservices/data-plane/Language/stable/2022-05-01/analyzeconversations.json:2077:10 ($.definitions.apimkey.properties.name[\"x-ms-enum\"])\nFATAL: swagger-document/individual/schema-validator - FAILED\nFATAL: Error: [OperationAbortedException] Error occurred. Exiting.\nProcess() cancelled due to exception : [OperationAbortedException] Error occurred. Exiting.\n"

[konrad-jamrozik]

Another case:

• https://github.com/Azure/azure-rest-api-specs/pull/25016/checks?check_run_id=15442460997
• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2958286&view=logs&j=8f628bfe-7d8d-5bf3-e141-37106862c4f5&t=723dfc11-d28f-5d74-45f1-663cde6746b4

[konrad-jamrozik]

Another case:

• https://github.com/Azure/azure-rest-api-specs-pr/pull/13923/checks?check_run_id=15390299761
• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2952865&view=logs&j=98f87804-8e1f-5655-af02-e80aefa7aa97&t=65457206-7ab6-5966-2a2f-9ae3efa14a03

The tool detected the breaking change, but then failed with runtime error, so it never produced the signal required for automation to add necessary labels.

[konrad-jamrozik]

Another case:

• https://github.com/Azure/azure-rest-api-specs-pr/pull/11374
• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=2933861&view=logs&j=98f87804-8e1f-5655-af02-e80aefa7aa97&t=65457206-7ab6-5966-2a2f-9ae3efa14a03&l=202

[konrad-jamrozik]

Another one:

• https://github.com/Azure/azure-rest-api-specs-pr/pull/13606/checks?check_run_id=16018267282
• Email thread: RE: Breaking change approval request - new APIs

This case denotes the runtime exception may possibly be caused by the tool trying to compare to a non-existent file.

[konrad-jamrozik]

• 6447

[konrad-jamrozik]

I did analysis and bucketing of some of the runtime errors. See here:

• https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_git/openapi-alps/pullRequest/500754#1695931281

[konrad-jamrozik]

So far big chunk of this work was addressed by:

• Pull Request 500754: Implement support for pretty-printing runtime errors originating from AutoRest invocation from within OAD (breaking change detector)

[konrad-jamrozik]

Closing for now. May revisit with future work later. For now we identified more urgent work.

[konrad-jamrozik]

Reopening for Dilithium