Allow extensions to publish beta releases and users to opt-in to them

[DanTup]

Apologies if this has come up before; I'm unable to find any related issues.

It would be cool to be able to publish extensions in a beta form (similar to Code Insiders) so that they can be run "in production" with real users that have opted-in. This would reduce the disk in publishing an extension because you can push it to a smaller number of users for testing (I probably don't need to sell this to you, I guess you have Insiders for very similar reasons).

I know we can package and distribute extensions ourselves for now, but it's not as easy (eg. updating is not automatic).

I originally thought the "Preview" flag might've provided this functionality but unless I've overlooked something, it does not seem to.

[brettcannon]

@DanTup for us it's more of a question as to whether we use this mechanism or publish a separate nightly extension for the next 18 months. We have a lot of users who sideload the Python extension, so having that "download" link in the MP point to something that isn't our stable release will be a support burden as people will inevitably open issues about getting our insiders build from there no matter how well we try to document where to actually download from.

[DanTup]

@brettcannon Yep, I understand. Presumably the proposal above doesn't prevent you from continuing as you are today, and consolidating things when the marketplace is updated.

For me, having a second extension would actually mean having a 3rd and 4th (Dart + Flutter - the Flutter extension has a dependency on Dart, so there couldn't be a Dart Insiders without also a Flutter Insiders) and I'd have to deal with users potentially having both sets installed. Having a single-extension solution would definitely be better for me even with the compromises above.

[isidorn]

Thanks a lot for feedback 👏 This week we plan to listen to your suggestions and to discuss this with the team. Next week we should have a definite decision if we will go down this road. Though let me try to answer some questions:

1. Good point by @eamodio to use different storage if extension is stable or insiders. This makes sense and if we implement the proposal @sandy081 and me will explore this
2. I have contacted the Marketplace team to see how much effort it would be for the Marketplace site to show the latest "non insider" version instead. I'll update here once I hear back from them.

[brettcannon]

@DanTup we just got feedback similar to your dependency explosion issue from @jakebailey as Pylance depends on our Python extension.

I expect we will go with whatever @isidorn and @sandy081 come up with and just be as clear as possible in our docs about where to download a stable VSIX to side-load (if the Marketplace team doesn't come back with "good" news for @isidorn 😉).

[sandy081]

All, please note the following change in the proposal

Given that MP supports publishing smaller/older versions (given that the version is not published already), Extension authors can still release insiders and stable (including patches) in parallel. One recommendation is to use patch for insiders versions but extension authors are free to use any numbering model given that they use only major.minor.patch semantics. Example:

• Assume that you have a stable released version 1.0.0 and want to release next insiders version.
• You can release next minor insiders versions as follows: 1.1.0, 1.1.1, ....... , 1.1.25 (marking them using `--insiders flag through vsce)
• You can promote above insiders version to stable version by releasing 1.1.26 and start working on next insiders versions - 1.2.0... just like above.
• If you want to fix a bug in previous stable ie., 1.1.*, you can fix the bug and release 1.1.27.

This will allow you to continue releasing insiders and at the same time you can release stable patches without affecting insiders version.

[isidorn]

We really appreciate the feedback and we will make a final decision next week whether to go ahead with the proposal. In the meantime two more questions that I would like to get your feedback on:

1. Should VS Code insiders users gets insiders extensions by default? Or the insiders users need to opt-in to the insiders extensions?
2. If a user is on an Insider extension, and a new Stable version of that extension is released that has a higher version than the insider version. Should the user be updated to the just released stable? In other words opting to insiders does the user always get highest version even if that version is stable or not?

[chrmarti]

1. This might depend on the target stability of an insiders extension. If that's more on the experimental side, opt-in makes sense, if it's on the release-candidate side, opt-out will work better because it ensures a larger number of users will get it.
2. Extensions might enable experimental features in insiders versions, therefore insiders should stay on insiders also when there is a more recent stable version. (This seems similar to VS Code Insiders.) We might need a way for extensions to say when the insiders channel is discontinued, otherwise users might end up installing an out-of-date version.

[DanTup]

1. I'd probably prefer them to get Insiders by default, but I think @chrmarti makes some good points. Even if they have to opt-in, that still seems better than what we have today (and the two-extension thing is ofc already opt-in).
2. I think Insiders should stay on Insiders to avoid confusion, but again I don't feel too strongly about this. It's likely after a stable release I'd be making an Insiders release with a higher version number anyway.

[gjsjohnmurray]

1. Maybe allow the extension to decide?

[rchiodo]

1. I'd prefer insiders by default. Opt-in means a lot less insiders users for us and more confusion when asking people to try out fixes. Right now we just say if you want to try this fix, install VS code insiders.
2. I think I'd prefer insiders stay on insiders for the reasons that @chrmarti explained. Insiders bits can sometimes have experimental stuff that we're only trying in insiders and even if it doesn't, all fixes that end up in stable are always in insiders first for us anyway (so even if stable is newer, insiders will quickly follow).

[sean-mcmanus]

1. Our extension would highly prefer VS Code Insiders to get Insiders extensions versions by default; otherwise, Insider usage would drop too much.
2. Our extension would prefer Insiders to use the highest version, whether insiders or non-insiders, since our non-Insiders with a higher version is always a newer version than the previous Insiders -- that behavior could be overridden by extensions by publishing an insiders with a higher version as discussed in https://github.com/microsoft/vscode/issues/15756#issuecomment-919784379 and "experimental" features that are not desired in the higher versioned non-Insiders could potentially be enabled via a setting instead of relying on a vsix version. This would allow our extension to publish 1.7.0-insiders, then 1.7.1, and then 1.8.0-insiders later on; otherwise, we'd have to publish 1.7.1 and 1.8.0-insiders simultaneously with identical bits, which would be unnecessary, although not a major issue.

A package.json setting to allow extensions to opt-in to the different update behaviors would be fine too.

[jakebailey]

1. For Pylance, it would be beneficial to have insiders automatically get insiders extensions, but this isn't something we have now, and I'll personally be happy with any sort of increase in this userbase, be it from it being auto-installed or the UI being extended to show that insiders versions are available (which is going to have to happen anyway). I can see this potentially causing an issue from the POV that people trying out VS Code insiders will not only get VS Code's unstableness, but any unstableness from every extension they have installed that also has insiders.
2. I don't know if it makes sense for extensions to be upgraded from insiders to stable; if storage is separated out as in https://github.com/microsoft/vscode/issues/15756#issuecomment-919359976, this will probably cause oddities as users move back and forth within the channels. For example, we might intend to show a popup once (then never again), except that moving between stable/insiders will effectively clear that status. In Pylance's case, every time we publish a stable version, we also automatically publish a new insiders build that is semantically newer (so 2021.9.2 is equivalent to 2021.9.3-pre.1), so I'm not too worried either way.

[sandy081]

I don't know if it makes sense for extensions to be upgraded from insiders to stable; if storage is separated out as in Allow extensions to publish beta releases and users to opt-in to them #15756 (comment), this will probably cause oddities as users move back and forth within the channels.

@jakebailey I do not think we would be having separate storages for stable & insiders of the extension. As of now we would like to go with current model - one storage per extension. Having two storages will be an advanced feature request going forward.

[sandy081]

Maybe allow the extension to decide?

@gjsjohnmurray Yeah, extensions can do that by using vscode engine property in package.json. They can force insiders to use insiders of an extension by setting vscode insiders version in vscode engine property of package.json. This is what Jupyter & Remote Containers extensions are doing currently.

[connor4312]

For (2) my vote is in the court that Insiders users be updated to Stable if the most recent stable version > insiders. This matches the semver model to avoid inventing new mechanics for VS Code's versioning. Perhaps the ExtensionContext has a updateChannel enum that lets the extension know whether this is a stable or insiders user in order to enable/disable features?

[CsCherrYY]

Overall, the proposal is very helpful for the Java extension since we need an insider version to expand the visibility of our beta features. For the two questions:

1. I suggest VS Code insiders users could get insiders extensions by default. Since Java extension doesn't have an insider version yet, this will help our insider extensions used by more users and they can also have a change to switch back to the stable versions if needed.
2. When --insiders becomes available, we plan to release a new stable version along with a new insider version to keep the newest semantic version of Java extension is always the insider version, just like what VSCode insiders and Pylance do (mentioned before). Thus, Java extension will not meet this problem, I'm OK with both choices.

[isidorn]

Thank you very much for your great feedback. After a milestone of house keeping and thinking about all of this @sandy081 @misolori and me plan to do the following this milestone:

• Continue to use the current major.minor.patch version semantics but provide a way to extension authors to mark a version as insiders by introducing a flag --insiders to vsce during a publish step
• If a user opts to insiders extension the user will always get the highest version even if that version is stable
• For VS Code Insiders users by default we will offer users to get the Insiders extensions (image attached). We will use a dropdown so users can easily choose to go with the stable version instead. On VS Code stable users can still choose but the default option will be to install the Stable version of the extension.
• If an extension has an Insiders version available we will always render the Green badge in the upper right corner to indicate this both in VS Code Stable and Insiders
• There will be an easy way for users to switch between Stable and Insiders versions of an extension. Both for VS Code Stable and VS Code Insiders
• We will provide migration support for extensions. For example, when the separate GitLens (Insiders) extension gets deprecated VS Code will automatically install the regular Git Lens extensions and switch it to the Insiders version. We will also transition the extension state. Extension authors will have to contact us for this explicitly
• Marketplace will render the latest non Insiders extension version on the Marketplace site

Default Experience

Switching to Insiders

[DonJayamanne]

@isidorn thanks for the update

We will provide migration support for extensions. For example, when the separate Git Lens Insiders extension gets deprecated VS Code will automatically install the Git Lens extensions and switch it to the Insiders version. We will also transition the extension state. Extension authors will have to contact us for this explicitly

The above didn't make much sense. Your are talking about migrating insider gitlense to insider.

[isidorn]

@DonJayamanne thanks. I have tried to clarify this by editing that sentence and by adding links to clarify which extension I am exactly talking about in what part of the sentence. Hope it is clearer now.

[CsCherrYY]

Thank you all for the great work. Here I have another question, can extension have a way to know whether it's an insider version itself in runtime? Thus, we can use different APIs easily.

[jks-liu]

Thank you all for the great work. Here I have another question, can extension have a way to know whether it's an insider version itself in runtime? Thus, we can use different APIs easily.

I don't think it's a good idea. Same extension with same version should has same behaviour, no matter insider or not.

I think for your case you can get version string in runtime, and use different API according only to the version.

[DanTup]

@isidorn sounds neat! I presume if we push a stable version that's higher than the last insiders version, everyone (including insider-extension-version users) will be upgraded to it, but VS Code will remember those that were on the "insiders channel" so if we then push another Insiders version, those users would update back to that?

[sandy081]

Yes, that's correct.

[eamodio]

@isidorn Thanks for the update.

I think I'm good with most of this, but my biggest concern is when a user transition from an insider's version back to stable. Thinking mainly of data and settings migrations that could have happened that could now cause issues with the stable version (leading to a broken or unstable "stable" version). For stored data this can be mitigated more easily by storing the data for stable vs insiders differently (e.g. scoping the key), but with settings it gets a bit trickier. Maybe this is mostly a non-issue as dealing with these issues would only arise in a small set of extensions, but it might be good to have some documentation around best-practices to handle/avoid them.

My other thought is around versioning, but I think it might be a non-issue (though I haven't thought through all the scenarios). Most Insiders/Nightly extensions I've see use a date/time versioning model, of which I think is highly valuable to both the extension and the user, and I think given the rules outlined above an extension could still utilize this versioning. Since Insiders versions using YY.MM.DDHHMM (or similar) would always be greater than the stable version, so you'd never get "upgraded" from Insiders to Stable (which imo can be desirable). Are there any other issues with try to keep that versioning scheme (if an extension wanted) that you all can think of?

A couple of thoughts/clarifications on the UI mocks:

• In the images of the 2 versions of GHPR -- they both seem to say they are the same version. I'm guessing (hoping) that was just a mockup issue, and if you were on version 1.5 on stable, and you switch to insiders (and the latest is 2021.11.151010) then 2021.11.151010 will be shown as the version in the UI?
• It seems redundant to have the "This extension has an Insiders version available" message when you are on the insiders version -- almost makes it feel a bit broken.
• I think there should be a badge or something applied to the extension in the list when you are running an insiders version -- easier to scan for -- I think it probably should warrant a badge on the extension overview page too
• I think it would be nice for extensions to be able to provide an insiders icon image if they want to further differentiate the editions
• I think the terminology of "Switch to [Insiders|Stable] Extension" on the button could be improved by calling them Channels (or Editions, though I much prefer channels). I think it could also help users better understand what they are opting into. Other platforms use the channels metaphor.

For VS Code Insiders users by default we will offer users to get the Insiders extensions (image attached). We will use a dropdown so users can easily choose to go with the stable version instead. On VS Code stable users can still choose but the default option will be to install the Stable version of the extension.

Can you show a mock of what this would look like for a first time install? Both in the list and in the overview page?

[isidorn]

@eamodio thanks a lot for the feedback. Let me try to answer your questions:

Yes, we should think more about what transition from Stable to Insiders extensions means for data and settings and we should document it. @sandy081 probably has more thoughts here

No, I do not see other issues with that versioning scheme.

1. Yes, this is a mockup issue
2. I agree. We'll improve this in the real version
3. There is a green circle badge in the top right which shows this. We can fine tune this
4. This is a fair feature request, not sure if this might be possible to do already if an extension publishes a different icon every time it publishes the insiders editions. I understand this is a manual process and we might need something more straight forward. Good feature request after we ship this
5. I like the word "channel". We could use "Switch to Insiders Extension Channel" / "Leave Insiders Extension Channel". Might be too long. We decided to not use the word "stable". @misolori for thoughts

First time install for VS Code Insiders would look something like this (mock still in polish phase)

[bobbrow]

@isidorn I've been thinking about the versioning more as we consider what it would be like to transition the C++ extension to this system. Previously, if our upcoming release was, say, 1.5.0, we would have 1.5.0-insiders, 1.5.0-insiders2, etc, and then finally 1.5.0 (stable). We don't do daily insiders releases and only start releasing them when new features are available for evaluation. The way VS Code determined which extension to install was based some kind of "greater than" check, so our developers in the insiders program would automatically be upgraded to "stable" when the evaluation period for a release had completed.

It occurs to me that with this new way if only x.y.z versioning is supported, we'd have to do versioning one of two ways:

x.y.0 is the first stable Ideally, when we release a new version, the patch number is .0. In order to achieve this, it seems we'd have to release insiders for x.y as x.(y-1).z. This might be confusing because it doesn't indicate that new features are becoming available since only the patch version is changing.

Example: Current stable release is 1.4.0. We release the first insiders as 1.4.1, second insiders as 1.4.2. When stabilized we release 1.5.0.

x.y.z is the first stable In order to ensure that our insiders are upgraded to stable at the end of the evaluation period, we would start releasing insiders as x.y.0, then x.y.1, x.y.2 for subsequent insiders releases. The first stable release would have a version number of x.y.z.

Example: Current stable release is 1.4.3. We release the first insiders as 1.5.0, second insiders as 1.5.1, etc. When stabilized we release 1.5.3 (or whatever the current patch number is).

The nice part about using the - in our versions currently is that it allows us to denote that new features are coming (increment the minor version number) and also release the first stable with a patch number of 0.

Can we continue to use - in our versions when publishing with --insiders, or how would you recommend we do our versioning going forward?

[eamodio]

@isidorn Thanks!

I like the idea of Switch/Leave the insiders channel -- I think you can shorten it to Switch to Insiders Channel / Leave Insiders Channel or even Join Insiders Channel / Leave Insiders Channel.

Alternatively, rather than buttons, this could be a "[Release] Channel" dropdown that has a Stable (or Release) and Insiders options.

But I think in any of these options there needs to be some explanatory text as to what an insiders version/channel is.

[sandy081]

• I do not think we would like to handle any data partition or migration for stable and insiders versions of the extensions. VSCode would treat this as a normal update to higher/lower version. We will make it clear about this in the doc and also recommend such extensions to use separate storage keys (append with insiders for eg) for differentiating.

• I think using Channel might send different message about this feature. Briefly, with this feature, VSCode is allowing users to install and use insiders/pre-release versions of an extension. For such users, VSCode will always install the latest version of the extension and this can be a pre-release or release version. So the name Join/Leave the Insiders Channel might convey that user will be always in Insiders channel even if there is latest stable. It is up to the extension to always have a latest insiders version and force users to be on insiders version but it is not mandatory. Yes, I agree that we have to improve on conveying this message in our UI.

• Using patch version for pre-release tags is a temporary solution because Marketplace does not support pre-release tags at present. We (VSCode & Marketplace) have plans to support pre-release tags (SemVer) going forward and once it is available you can start using pre-release tags and follow the same release cycle as you are using now, ie., 1.5.0-insiders -> 1.5.0-insiders2 -> 1.5.0 without any big adoptions/changes and it will work seamlessly. Until then we recommend to use patch part for pre-release/insiders versions as mentioned in this comment

CC @eamodio @bobbrow @isidorn

[bobbrow]

Thanks @sandy081

[Lemmingh]

Briefly, with this feature, VSCode is allowing users to install and use insiders/pre-release versions of an extension.

https://github.com/microsoft/vscode/issues/15756#issuecomment-970482005 https://github.com/microsoft/vscode/issues/15756#issuecomment-971450154

What about still having just one "Install" button and adding a "Receive Insiders" switch next to it?

Install | ☑ Receive Insiders

This extension has an Insiders version.

Disable | Uninstall | ☑ Receive Insiders

You are receiving an Insiders version. This extension is enabled globally.

Disable | Uninstall | ☐ Receive Insiders

An Insiders version is available. This extension is enabled globally.

[isidorn]

@Lemmingh thanks for your feedback, however we have decided on the current design, we are currently in the "polish wording" phase.

[isidorn]

We have pushed this feature to Insiders 🚀
We are in the process of testing it. If all looks good end of this week I will write docs for extension authors on how to best adopt to this and I will share them here. GitHub Pull Request extension adopted to this, so for those who are eager to try out the flow can use VS Code insiders and that extension to see how it looks.

[isidorn]

We have released a VS Code stable that supports this 🚀 Here are the extension author docs that should help you adopt to pre-release extension support https://code.visualstudio.com/api/working-with-extensions/publishing-extension#prerelease-extensions

If there are any questions do not hesitate to reach out. Looking forward to feedback. Thanks a lot 👏

[DanTup]

@isidorn awesome! Two questions:

• what happens if we publish a pre-release version now, for users that haven't upgraded VS Code yet? Would they see the pre-release version as an update, or would it be filtered out on the server?
• in a comment above you said "Marketplace will render the latest non Insiders extension version on the Marketplace site" but on the page at https://marketplace.visualstudio.com/items?itemName=GitHub.vscode-pull-request-github it shows Version | 0.35.2021120909 which looks like an insiders version (I see 0.34.0 in VS Code as the stable version). Is this a bug, or has the marketplace not been updated yet?

[isidorn]

@dantup good questions:

• vsce will require you to specify an engine property in your pre-release version so that older VS Code versions will not be able to use it. So users that haven't upgraded will not be offered the pre-release version
• Correct, MP has not yet done this work. This item tracks the work here https://github.com/microsoft/vsmarketplace/issues/264 In the end we decided to have more UI decorations to make it clearer to the user that it is a pre-release version. We should have something on the MP side by the time most of the extensions adopt to this (end of January). Let's continue the discussion on that issue for this.

[DanTup]

vsce will require you to specify an engine property

Ah, perfect. Thanks! :-)

[CsCherrYY]

@isidorn Thanks for the hard work. We're looking forward to a way to publish Type Hierarchy feature as a preview feature and pre-release extension is a great choice. However, the related APIs are in proposed state (see https://github.com/microsoft/language-server-protocol/pull/1231#issuecomment-929296135 and https://github.com/microsoft/vscode-languageserver-node/blob/main/client/src/common/proposed.typeHierarchy.ts), enabled by enabledApiProposals and we can't publish it even to the pre-release channel.

The question is that we want to find a way to take advantage of the pre-release channel to make more users use our preview features with proposed APIs, is that possible? Or is there any consideration about this?

[isidorn]

@CsCherrYY there have been some discussions around this but nothing definite yet. We want to make sure that the pre-release extension support is adopted smoothly across our community before we decide to build any additional features on top.

[davidanthoff]

Is there some way to find out from code whether the currently running extension was installed as a pre-release version or not? We want to send telemetry to different destinations, depending on whether a user is running a pre-release version or not.

[TwitchBronBron]

Is there some way to find out from code whether the currently running extension was installed as a pre-release version or not? We want to send telemetry to different destinations, depending on whether a user is running a pre-release version or not.

A version cannot be both pre-release and not pre-release, so if you manage the extension, then you are in control of the process for launching pre-release versions and therefore know which versions are pre-release. If you follow the suggestions outlined in the documentation:

we recommend that extensions use major.EVEN_NUMBER.patch for release versions and major.ODD_NUMBER.patch for pre-release versions. For example: 0.2. for release and 0.3. for pre-release.

then you can easily check the current extension version and act accordingly based on whether is an even or odd minor release.

[isidorn]

@davidanthoff exactly as @TwitchBronBron explained: you can look at your extension version and act accordingly.

[rchiodo]

There's a lot to read through in this issue and I wasn't sure what the expected behavior is with regards to 'insiders'.

Should insiders users automatically be updated the prerelease of an extension?

This isn't currently happening for the Jupyter Extension.

I had to explicitly pick 'Switch to prerelease version'

[sean-mcmanus]

Uh oh...if VS Code Insiders users have to manually opt-in, we're likely to see our Insiders usage drop by a lot. @bobbrow Are you sure we should switch to the new mechanism if that's the case? And/or can we get VS Code to change the behavior first?

[isidorn]

We support automatically migrating separate extensions to the pre-release version of the main extension - extension authors just have to inform us they want this. However since C++ and Jupyter have a special case and you do not have separate extensions we currently do not support automatic migration done by VS Code. However you can do the migration on your own as discussed here https://github.com/microsoft/vscode-cpptools/issues/8507#issuecomment-994178165

[gjsjohnmurray]

For me just now, when I used Insiders and searched for the GHPRI extension (which I know published pre-releases), the Install button defaulted to Pre-Release:

Doing the same with Stable defaults to a regular install:

[rchiodo]

We support automatically migrating separate extensions to the pre-release version of the main extension - extension authors just have to inform us they want this. However since C++ and Jupyter have a special case and you do not have separate extensions we currently do not support automatic migration done by VS Code. However you can do the migration on your own as discussed here microsoft/vscode-cpptools#8507 (comment)

@isidorn I'm not sure that's going to work? Won't it force all users to the latest? For me I didn't want the latest, but when I do, it should show me the latest is the prerelease version. That's the behavior we have right now. Users pick when they upgrade, but the recommended upgrade is the most recent insiders only build.

[isidorn]

@gjsjohnmurray correct. @rchiodo yes, the insiders users will by default get the option to install Pre-Release. My previous answer was my misunderstanding of your question. For me this just works

[isidorn]

@rchiodo you would just do that code for your insider version of the extension. So that way you would transition only your insider users to the new pre-release version.

[rchiodo]

@isidorn that's the behavior with the jupyter extension not already installed. But if it was already installed, the update notification won't pick prerelease.

And I believe the command suggested would force an update? Maybe I'm wrong about that.

[rchiodo]

I guess it won't be terrible to force prerelease if the extension isn't already in insiders for a specific version (one time thing).

Maybe we can add code to ask the user on insiders.