Kiota Coverage Test Suite

[darrelmiller]

Using the GitHub search mechanism, walk the advertised OpenAPI descriptions, generate clients and compile those clients. We can exclude Microsoft Graph because we already test that elsewhere

Benefits:

• This will give us advanced visibility into OpenAPI descriptions that are index but don't work with Kiota
• This will give us metrics on the number of APIs index
• This will provide confidence about our support for broad coverage of published APIs
• Using the warnings and errors we detect about Kiota's ability to support certain constructs, we can prioritize where to enhance support.

[darrelmiller]

Example of code to walk the Apis.guru list https://github.com/microsoft/OpenAPI.NET/blob/vnext/test/Microsoft.OpenApi.SmokeTests/ApiGurus.cs#L41

[sebastienlevert]

Some thoughts on the APIs we should be testing against. Using this https://www.postman.com/explore/most-popular-apis-this-year to identify the most used. I'm linking either the yaml file if available, or the Postman collection, from which we could export the collection and transform it to OpenAPI.

• Salesforce Platform APIs (https://postman.com/salesforce-developers/workspace/salesforce-developers/collection/12721794-67cb9baa-e0da-4986-957e-88d8734647e2?ctx=documentation)
• Twitter API v2 (https://api.apis.guru/v2/specs/twitter.com/current/2.21/openapi.yaml)
• Notion Public Beta (https://api.apis.guru/v2/specs/notion.com/1.0.0/openapi.yaml)
• WhatsApp Cloud API (https://api.apis.guru/v2/specs/whatsapp.local/1.0/openapi.yaml)
• Microsoft Graph (https://raw.githubusercontent.com/microsoftgraph/msgraph-metadata/master/openapi/v1.0/openapi.yaml)
• Slack Web API (https://api.apis.guru/v2/specs/slack.com/1.7.0/openapi.yaml)
• PayPal APIs (https://postman.com/paypal/collection/19024122-92a85d0e-51e7-47da-9f83-c45dcb1cdf24)
• Stripe (https://api.apis.guru/v2/specs/stripe.com/2020-08-27/openapi.yaml)
• Google Maps Platform (https://raw.githubusercontent.com/googlemaps/openapi-specification/main/dist/google-maps-platform-openapi3.yml)
• Razorpay APIs (https://postman.com/razorpaydev/workspace/razorpay-public-workspace/collection/12492020-952c7295-118c-400f-8f2c-5266ef6f689a?ctx=documentation)
• MongoDB Data APIs (https://postman.com/mongodb-devrel/workspace/mongodb-public/collection/17898583-25682080-e247-4d25-8e5c-1798461c7db4?ctx=documentation)
• Zoho CRM REST APIs (https://postman.com/zohocrmdevelopers/workspace/zoho-crm-developers/collection/8522016-0a15778a-ccb1-4676-98b7-4cf1fe7fc940?ctx=documentation)
• Datadog (https://postman.com/datadog/workspace/datadog-s-public-workspace/collection/7274195-66ef21d8-e159-4d7d-8ded-c511e1abe189?ctx=documentation)
• Meraki Dashboard API (https://api.apis.guru/v2/specs/meraki.com/1.12.0/openapi.yaml)
• Pipedrive API (https://developers.pipedrive.com/docs/api/v1/openapi.yaml)
• Twilio Messaging / SMS (https://api.apis.guru/v2/specs/twilio.com/api/1.38.3/openapi.yaml)
• Amplitude Analytics APIs (https://postman.com/amplitude-dev-docs/workspace/amplitude-developers/collection/20044411-88bd70c1-9e00-483c-aaf1-2b7159f9bb2b?ctx=documentation)
• DocuSign REST API (https://api.apis.guru/v2/specs/docusign.net/v2.1/openapi.yaml)
• BookingAPI (https://postman.com/clientapi/workspace/apitude-by-hotelbeds/collection/295910-c411a5fa-523c-4048-afef-5afd52a294b4?ctx=documentation)

Now, these would be considered as "most popular" but probably not diverse enough to help us identify edge cases. Should we just loop through all of APIs guru catalog and generate every possible clients?

[darrelmiller]

It's what speakeasy did. https://github.com/speakeasy-api/openapi-directory/tree/main/SDKs I have mixed feelings about it.

[sebastienlevert]

I absolutely agree with you. I don't think it's necessarily adding real value... Using the top APIs is certainly a good first step, but I don't feel we will find all the crazy things we can see in an OpenAPI description... @baywet or @RabebOthmani any idea on how we can get a high level of confidence we are generating valid clients with a diverse set of API definitions?

[baywet]

As I've started working on this, I did have a couple of remarks:

• I don't think using msgraph in the smoke tests set brings much value as we're designing with Microsoft Graph in mind and already have extensive processes for quality (raptor etc...)
• I don't think going and getting every random API definition out there is going to provide much ROI. There are duplicates (multiple flavors of Azure, GitHub, etc...) and the triage process is going to take much longer for each new API definition we use in the test set.

Bottom line, starting with 15/20 popular APIs like Seb listed is a good starting place. I do have a couple of remarks on that list:

• Is it worth including the postman collections? If we identify defects of the description, we have nowhere to report them.
• Should we include Twitter, given what's currently going on with their API story?

Early implementations already uncovered a couple of things #2351, #2352, #2353 (and that's without counting the issues I've reported to GitHub directly)

[sebastienlevert]

I agree for Twitter, we might want to hold on it. Though, it's a very popular API so with (or without) the paid APIs, I think it's still relevant from a generation perspective.

For Postman collections, you might be right... It makes 11 OpenAPI descriptions... Any other candidates we want to include?

[baywet]

Adding to the list, a lot of descriptions will fail because of the lack of union/intersection types support in TypeScript (#1812) and other languages.

[sebastienlevert]

Failing now, or failing later? We expect to support these scenarios right?

[baywet]

failing now, union type support was never implement in typescript because that was at the time we rebooted efforts to have a lighter experience in typescript and I wanted to avoid potential conflicts. As per python/php, some of the work is still ongoing. And it remains to be done entirely in Ruby

[baywet]

Note from the meeting:

• skip metrics for now
• kill the Data and type mismatch found. validation rule (see GitHub API runs)
• in a separate process, collect the warnings, do not build the result. map and reduce the warnings. use a large dataset (create a separate issue for that?)

[baywet]

APIs ignored:

• whatsapp: hasn't been updated in 2+ years
• slack: not updated in 2+ years
• (all postman collections)
• Microsoft Graph

new bugs uncovered

2360, #2361, #2362

[baywet]

additional bugs

2366, #2367, #2368, #2369

[sebastienlevert]

Really happy to see these bugs being caught!

[baywet]

additional issues

2372, #2373, #2374, #2375, #2378