search

Azure / azure-sdk-tools

Tools repository leveraged by the Azure SDK team.
MIT License
113 stars 177 forks source link

Develop tooling / dashboard to measure / report "freshness" of client libraries #3303

Open mikekistler opened 2 years ago

mikekistler commented 2 years ago

A service team must request an update to client libraries when it releases new features, but if it fails to do so then customers cannot access the new features using the client libraries. We should monitor the "freshness" of client libraries to ensure that whenever the service API is updated the client library is also updated to support this new version.

kurtzeborn commented 2 years ago

Is this possibly solved by adding a "service version" column to this table: https://azure.github.io/azure-sdk/releases/latest/ where each library has a list of (or maybe just the latest) version(s) supported by a given sdk library.

mikekistler commented 2 years ago

I think that is a piece of the solution. Then we need to check that latest version supported by the client library against the latest version supported by the service, and raise an alert when these diverge for longer than say a couple weeks.

weshaggard commented 2 years ago

@mikekistler lets chat more about this on our sync-up this week as I agree we need to figure out how to detect this but it is a matter of where the right place is to put in those checks.

lfraleigh commented 2 years ago

While we should do this anyway, this explicit discussion started b/c of this issue the field raised: https://dev.azure.com/unifiedactiontracker/Technical%20Feedback/_workitems/edit/71540

josefree commented 1 year ago

@ladonnaq and @ccbarragan for the dashboard requirements

ladonnaq commented 7 months ago

@maririos @konrad-jamrozik I am working on cleaning up the backlog to prepare for next semester. I have a few questions about this issue and ideas.

  1. Is there something in place to remind a service team when the PR is merged that they need to now access the release plan to use the SDK release milestone app to guide them through the release of the SDKs? If they use the SDK release milestone app it will guide them thru the process and ensure the new SDK version is released.
  2. Is the automation (bot whatever) running a check the latest refresh of the SDK against the latest PR that merges. If so, we could store the data some place and if the new SDK version is not released within a specific timeframe (30-day window should provide enough time for generated SDKs) we alert the user that they need to release their new SDK versions.
maririos commented 7 months ago

Is there something in place to remind a service team when the PR is merged that they need to now access the release plan to use the SDK release milestone app to guide them through the release of the SDKs? If they use the SDK release milestone app it will guide them thru the process and ensure the new SDK version is released.

Is the automation (bot whatever) running a check the latest refresh of the SDK against the latest PR that merges

That I am aware of, we don't have automation that has data to check API spec published version and API spec used in SDK repo. This would be a greater effort. Today I saw this: https://github.com/Azure/azure-sdk/issues/1198 which touches on what you are asking

konrad-jamrozik commented 7 months ago

@ladonnaq, what @maririos said :)

In my mind, conceptually, we distinguish two scenarios:

  1. a specs PR was created by a PR author because release planner guided them to do it
  2. a specs PR was created by a PR author completely independently of release planner and they may not even know release planner exists

In case of 1. for now I believe we could continue making sure that the release planner knows about the PR, but the PR does not know about the release planner, at all.

In case of 2., as @maririos said, we could, as just one idea, design some kind of capability where the specs PR says something to the effect of "hey PR author, I see you created me without using release planner. Are you suuuuuure this is what you want to do?". I would have hard time prioritizing such kind of work any time soon, unfortunately.

ladonnaq commented 3 months ago

I am working on the extended roadmap for the engagement experience. I want to determine if this requirement should be added to the roadmap.

@sandeep-sen - Is this requirement associated with the work you were doing to determine the freshness of the SDKs and reach out to service partners to have them release new SDK versions?

@ronniegeraghty - How is the inventory dashboard determine freshness? If the data is obtained programmatically, we could ingest the data into the engagement experience dataverse and link the data with the service & product. Then from there we can decide how to make actionable. For example, we could extend the Service Partner dashboard to include a view for of actions for "SDKs need refreshed".

ronniegeraghty commented 3 months ago

@ronniegeraghty - How is the inventory dashboard determine freshness? If the data is obtained programmatically, we could ingest the data into the engagement experience dataverse and link the data with the service & product. Then from there we can decide how to make actionable. For example, we could extend the Service Partner dashboard to include a view for of actions for "SDKs need refreshed".

It's a little arbitrary. The freshness in the inventory dashboard is the number of days since the last release. The hope was to have it represent something like what is being discussed here. Are the SDKs not in sync with the API Spec. But there was no existing mapping between Spec versions and library versions for the dashboard to tap into.

ladonnaq commented 3 months ago

[like] LaDonna Quinn reacted to your message:

From: Ronnie Geraghty @.> Sent: Wednesday, July 17, 2024 4:36:40 PM To: Azure/azure-sdk-tools @.> Cc: Comment @.>; Assign @.> Subject: Re: [Azure/azure-sdk-tools] Develop tooling / dashboard to measure / report "freshness" of client libraries (Issue #3303)

@ronniegeraghtyhttps://github.com/ronniegeraghty - How is the inventory dashboard determine freshness? If the data is obtained programmatically, we could ingest the data into the engagement experience dataverse and link the data with the service & product. Then from there we can decide how to make actionable. For example, we could extend the Service Partner dashboard to include a view for of actions for "SDKs need refreshed".

It's a little arbitrary. The freshness in the inventory dashboard is the number of days since the last release. The hope was to have it represent something like what is being discussed here. Are the SDKs not in sync with the API Spec. But there was no existing mapping between Spec versions and library versions for the dashboard to tap into.

— Reply to this email directly, view it on GitHubhttps://github.com/Azure/azure-sdk-tools/issues/3303#issuecomment-2233734384 or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQEFYORQKUMUME7TZNW2GP3ZM2MRTBFKMF2HI4TJMJ2XIZLTSSBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVI2DEOBVGA2DKNBXGGSG4YLNMWUWQYLTL5WGCYTFNSWHG5LCNJSWG5C7OR4XAZNMJFZXG5LFINXW23LFNZ2KM5DPOBUWG44TQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKJRG4YDKOJSGE4DNAVEOR4XAZNFNFZXG5LFUV3GC3DVMWVDCMRTGAYTINRYGAZYFJDUPFYGLJLMMFRGK3FFOZQWY5LFVI2DEOBVGA2DKNBXGGTXI4TJM5TWK4VGMNZGKYLUMU. You are receiving this email because you commented on the thread.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

lfraleigh commented 2 months ago

In the last 24 hours, both @josefree and @lmazuel have come across teams that released new API versions but did not release corresponding SDK libraries. We need to both prevent this as well as catch it after the fact (as there's a lot of clean up we're probably going to need to do with teams).