[Question] Guidance on how to keep versions updated and aligned

[andreaTP]

Hi all 👋 I have been experimenting with various kind of Kiota integrations and something that I found challenging from time to time is how to keep versions in proper sync. Those reflections come from another iteration on a Python SDK.

Those are the issues:

• Kiota version itself - updating a command line tool without a package manager is already challenging and not supported by things like dependabot out-of-the-box - I just realized that #2589 got closed, but only the "shallow mitigation" done with asdf got merged there
• Kiota library versions - here we have various "source of truth"
  • dependabot - would always try to bump to latest even if Kiota(the generator) has not been upgraded, and some changes might depend on generator updates
  • kiota info - the command line is hardly usable by non-human (can be fixed by something like #3531 ) but the issue is that it won't reflect PATCH versions bump for e.g. security-related fixes

All in all, I see that most of the tooling I'm building is based on partial workarounds to those issues, and I believe that there is no "structural way", at this point in time, to have a "completely correct" or, at least, officially supported solution.

Questions:

• have you ever thought about releasing a BOM-like artifact?
• how do you handle this situation in e.g. Graph repositories? is it all manually curated?
• I recall that there were officially documented OpenApi extensions for Kiota, can they be used to mitigate the issue?
• do you have alternatives I have not considered?

[baywet]

Thanks for starting the conversation here.

I think we had a similar conversation a little while ago, but I can't find it anymore. Let's start by acknowledging this is particularly painful with languages that are not stable because we might change the public API surface of those packages, and the generator might be behind or ahead. We're trying to coordinate the best we can but the last few days have been a bit hectic getting ready to move PHP and Python to stable. Apologies for any inconvenience.

This can still happen for stable languages in case when the new version adds a new API the generated code will depend on, and your version of the generator is ahead of the dependencies you should have. (kind of the opposite of what you've experienced). This is why the first part of the recommendation is to use the same exact dependencies versions that were shared by the kiota info of the same version of kiota you used to generate the client the last time you did so.

Now, this leaves a gap for bugfixes and whatnot: we could release a patch on for a stable language dependency, and it could take a while before the next minor of kiota. I think we might want to update our guidance to add something like "you may try to update dependencies, but this could have side effects".

We were expecting a ton of feedback on the support policy, and dependencies version management, and even we the growth we're seeing (better than we could have wished for), we didn't get much feedback on either. Could have we gotten it right for the majority of people?

have you ever thought about releasing a BOM-like artifact?

This is something @darrelmiller kept mentioning a while ago and we have longer terms plans along those lines. But nothing to share publicly yet, and we're in the middle of our planning cycles for the next semester. We should definitively talk about that on the next community sync.

how do you handle this situation in e.g. Graph repositories? is it all manually curated?

We're bleeding edge sync the first goal of kiota was to modernize our SDKs story. But we recognize we're the exception and most teams won't be able to monitor dependencies alignment that closely.

I recall that there were officially documented OpenApi extensions for Kiota, can they be used to mitigate the issue?

https://github.com/microsoft/OpenAPI/blob/main/extensions/x-ms-kiota-info.md Kind of, you could emit your own "barrel" to control the dependencies versioning in a centralized way, and point to that in the extension.

do you have alternatives I have not considered?

Not really at this point.

[andreaTP]

This is why the first part of the recommendation is to use the same exact dependencies versions that were shared by the kiota info of the same version of kiota you used to generate the client the last time you did so.

ok, got it, having a machine-readable output of the kiota info command can help automate this aspect 👍 Probably we should also document how to exclude the Microsoft dependencies from bots like dependabot and everything should flow from the CLI.

Now, this leaves a gap for bugfixes and whatnot: we could release a patch on for a stable language dependency, and it could take a while before the next minor of kiota. I think we might want to update our guidance to add something like "you may try to update dependencies, but this could have side effects".

I'm not entirely satisfied with this, as it requires manual intervention and review of the process, and hence cannot be automated. Having to go through the described process for each and every library dependency bump is going to be a time sinking activity 😢 A possibility would be to force patch releases of the CLI just to bump the libraries, likely, this can be automated on your side and would offer a reliable way to identify the correct set of dependencies. wdyt?

The short-coming of using https://github.com/microsoft/OpenAPI/blob/main/extensions/x-ms-kiota-info.md is that it cannot be understood by tools like dependabot.

[baywet]

We already have a script that updates the dependencies for the info command on every release. Is that what you were suggesting?

extension cannot be understood by dependabot

Sure it reduces the problem of 6 or so dependencies down to one. It'd be an improvement but not a solution.

I'm not entirely satisfied with this

Fair enough, another alternative I can see for stable languages would be to strictly observe semver (not always the case historically) and advocate to use patch version ranges with kiota patch versions (e.g. kiota 1.8.0 works with kiota abstractions 1.5.X for dotnet), but it's not possible for every language (e.g. Go).

[andreaTP]

We already have a script that updates the dependencies for the info command on every release. Is that what you were suggesting?

Yup 👍 , unfortunately, it won't work for backports on stable releases, e.g. it won't understand to bump to 5.0.1 if 6.0.0 is already available.

another alternative I can see for stable languages

I don't think this proposal actually tackles the issue we are discussing, do you wanna elaborate further?

I think that, with the current decisions/constraints, a solid process for backporting library versions bump and releasing from maintenance branches in this repo is the best course of action on this one.

[baywet]

e.g. it won't understand to bump to 5.0.1 if 6.0.0 is already available.

and even if we updated it, it wouldn't know : kiota 1.8.0 needed dotnet abstractions 1.5.0, kiota 1.9.0 abstractions 1.6.0, but now we're releasing kiota 1.8.1, and it needs abstractions 1.5.2, even though 1.6.0 exists. I'm not sure we'll ever be able to fix that except if we use some kind of tag on the dependencies (e.g. the dotnet package would have a tag kiota1.7.0, which would be updated depending on which version it's been tested for), or some kind of gigantic mapping file (both approaches being time consuming and error prone).

do you wanna elaborate further?

kiota 1.7.0 up to kiota 1.7.9999999 would document needing abstraction 1.5.X instead of 1.5.0, and people would add it with a range (in case of dotnet [1.5.0,1.6.0) javascript 1.5.X, etc... )

[andreaTP]

kiota 1.7.0 up to kiota 1.7.9999999 would document needing abstraction 1.5.X instead of 1.5.0, and people would add it with a range (in case of dotnet [1.5.0,1.6.0) javascript 1.5.X, etc... )

this is simple, can be automated, and works for me as long as we can apply the pattern to all languages 👍

[baywet]

I'm pretty sure this is not possible in Go unfortunately... Not sure where to go (pun intended) from here.

[andreaTP]

I don't understand how it can't work with Go, from what I understand there is not such a limitation and branches and tags are fully supported: https://stackoverflow.com/a/42766877/7898052 and: https://labix.org/gopkg.in

Can you point out the specific limitation you are talking about?

[baywet]

using tags

yes, tags allow you to specify a version, but not a range as far as I understand.

using local clones

This would represent a lot of overhead for kiota users.

Gopkg.in

This approach reroutes people to a different "proxy" than the official go package management proxy. This proxy seems to be hosted and maintained by a single individual (who seems to work at canonical). I'm not sure we'd want to have every go kiota user take such a dependency.

[andreaTP]

I'm not an expert, but, using tags and branches we should be able to provide e.g. a 3.X ( as branch name) and 3.0.1 (as tag name) version.

[baywet]

aaaah I didn't think of that. So we'd be "rolling forward" the tags? Kind of like docker repositories do for images with the latest tag? every time we have a new release we'd replace the major and minor tags, e.g 3.X and 3.1.X

I think that could work assuming the go proxy doesn't block it for security reasons, it'd add some burden for us but I believe it should be automatable. Also it'll probably be unusual to most go developers, more discussions needed here.

[andreaTP]

Rolling forward is a possibility, using branch names another. Yup, it would need a little bit of work to connect the dots, but looks pretty much feasible.

[baywet]

Adding some context from the community call today. Locking all dependencies versions to align them with the kiota version number is probably too rigid and will add a high cost to out of band patches. The pressure for alignment will decrease over time as languages become stable, and as misalignements are less of a problem. But that issue will rise again as we work on a v2. A good middle ground could be to publish rollup/bundle packages for each language, the version of which would be in sync with kiota.

[EricWittmann]

Hey everyone. I just wanted to add my recent experience with the Typescript generator here. I'm working on a new project and using Kiota to generate both Java and Typescript clients. Overall it has been successful, but recently the typescript client broke because the abstraction libraries were auto-upgraded by dependabot. This resulted in the generated code not building, presumably because the abstraction interfaces changed in a non-breaking way.

Of course, both Java and typescript are non-GA, so I'm not surprised that I will experience this sort of thing from time to time. But it did take me longer than I expected to get back to a working version. I tried upgrading to the latest preview release of Kiota but that didn't work (I was suprised). Then I tried backing out and it took a few tries to find the right combination of CLI and abstraction libraries.

Also note that the CLI for Typescript did not print out versions of the abstraction libraries it expects/requires. It prints out which libraries you need, but not the versions of them.

I imagine this will be more stable once Typescript is GA'd, so that's fine. But I will definitely be interested to see how this ends up working in practice in the future.

In the case of Typescript, I'm thinking about using my CLI wrapper to transitively bring in the abstraction libraries:

https://www.npmjs.com/package/@kiota-community/kiota-gen

It would mean that kiota-gen would need a new release every time there was a new CLI release or a new abstraction release. But that seems OK. It would make the downstream/consumer experience much nicer. It would also mean that the CLI wrapper would be a dependency rather than a devDependency so I will need to think about that.

Perhaps you are right - a BOM is what is needed somehow. The problem is that the BOM/bundle would need to somehow have both the CLI and the abstractions. Will need to explore this!

[baywet]

Thanks for the great feedback here!

Versions not being printed: I think #3531 introduced a regression where the string format is not using the install command anymore. My suggestion:

• switch this to a table instead so we ALWAYS print the versions
• add the install command at the bottom of the table for information purposes

(I'll create an issue later regarding that)

TypeScript is still experimental, which is a lower stability than Java being preview. Hence the more frequent breaking changes. But we're working hard to close the loop on the last few major changes and move it to preview soon.

Thank you for starting this project, if you think it'd help getting traction, don't hesitate to PR the install page