Allow extensions to publish beta releases and users to opt-in to them

[DanTup]

Apologies if this has come up before; I'm unable to find any related issues.

It would be cool to be able to publish extensions in a beta form (similar to Code Insiders) so that they can be run "in production" with real users that have opted-in. This would reduce the disk in publishing an extension because you can push it to a smaller number of users for testing (I probably don't need to sell this to you, I guess you have Insiders for very similar reasons).

I know we can package and distribute extensions ourselves for now, but it's not as easy (eg. updating is not automatic).

I originally thought the "Preview" flag might've provided this functionality but unless I've overlooked something, it does not seem to.

[ramya-rao-a]

I would very much like this myself.

[joaomoreno]

Why not putting those experimental features behind a configuration setting and have the opt-in mechanism be toggling that setting to true? Once the feature is stable enough just remove the setting.

[DanTup]

Why not putting those experimental features behind a configuration setting and have the opt-in mechanism be toggling that setting to true?

This is an option, but I think it has some drawbacks

1. If you want to do a beta release every day, your "stable" users will get the update notifications too
2. It doesn't eliminate all the risk (if you're shipping bugs in the new features, maybe your feature detection is buggy, or how you're dealing with two code paths)
3. For things that aren't just new features, it's more work to maintain/switch between the branches
4. Users have to keep opting-in to each feature, rather than choosing to be a beta tester once; this could really limit the testing you get
5. Having the feature toggles show up in the config auto-complete isn't ideal (this might not be an issue, but I didn't get a response to https://github.com/Microsoft/vscode/issues/1396#issuecomment-244350985 so it's unclear if "hidden options" are supposed to be used)

None of these issues are huge, but since the Insiders builds work so well and it seems like a similar scenario it seemed like a good request to raise for discussion :)

[davidanthoff]

I agree, this would be a really useful feature. Right now I'm distributing our beta releases via VSIX downloads, but that is cumbersome and I don't reach many folks that way.

Hiding things behind a feature flag wouldn't really help. For example, we are about to release a new version that changes a lot of code, the whole architecture of the extension. There is no way that we could have had that behind a feature flag.

[eamodio]

I would love to see this as well -- while feature flags work in certain cases, it doesn't cover them all. It also provides a sanity check and limits the "damage" of a bad release. And installing via vsix is just too user unfriendly to really be viable imo -- as it is completely manual.

[eamodio]

Just to kick this one again, because the issues surrounding the manual install and REQUIRED uninstall of vsix causes lots of pain for users and seemingly creates cases of corrupt installations -- I've definitely seen an up tick in GitLens issues caused by vsix/marketplace versions on top of one another.

[DanTup]

Yeah, I meant to post back here - when I did a beta release a few months ago I had reports of issues installing/uninstalling the vsix:

• https://github.com/Dart-Code/Dart-Code/issues/318#issuecomment-308244455
• https://github.com/Dart-Code/Dart-Code/issues/318#issuecomment-308263912
• https://github.com/Dart-Code/Dart-Code/issues/318#issuecomment-308279628

[DanTup]

I gave up on vsix testing because of the complications mentioned above. Yesterday I shipped a huge bug that only occurred if dart on your PATH was a symlink (I had to make a lot of changes for multi-root support, and putting it all behind flags was not feasible) :(

I'm starting to think about publishing a second version of my extension with Beta in the name to allow people to choose to test new releases earlier (seems like at least one other has done this).

[sandy081]

@DanTup @eamodio We did some improvements in this milestone (November) to support beta releases in VS Code when installed through VSIX. Refer to this issue https://github.com/Microsoft/vscode/issues/39024 for more details. Can you please check if that helps you in using VSIX?

[DanTup]

@sandy081 Sounds good! Am I right in thinking november milestone is 1.19 and not released to stable yet? I'll try and do some testing latest this/next week.

[sandy081]

Yes 1.19 will be released to stable this week. Let me know if you still want this feature to be opened after that.

[DanTup]

@sandy081 Haven't had chance to test it yet; but I think there's still value in this feature even if vsix works better than it did, for example:

• I think end users would have more confidence installing a beta vsix through the marketplace then from some random download url
• Allowing people to opt-in to a beta channel would allow them to get auto-updates of betas instead of having to do manual installations all the time
• Listing betas on the marketplace is much more discoverable - only a fraction of my users will follow the twitter account (etc.), so there are a lot of potential testers that simply won't know about beta vsix releases

[sandy081]

@DanTup All your requirements are valid. But I think this has to be supported by the Market place first.

@modigrv May I know if there are any plans to have this support in Market place?

[vscodebot[bot]]

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

[DanTup]

😔

[ramya-rao-a]

Re-opening this issue as we haven't reached any conclusion yet. ping @seanmcbreen and @modigrv on the question on marketplace

[sean-mcmanus]

Our C/C++ extension wants to be able to publish insider builds to the marketplace so users who opt-in can automatically get either the "insider" version or the release version after that's published -- mostly to increase our real-world test coverage before pushing changes to everyone. We can publish the insider vsix to our GitHub site and email users about it who sign up for it, but it requires user to manually install it, so that isn't likely to get enough usage. We're fine with all VS Code Insiders users also getting the Insider extension instead of having a separate insider setting just for our extension.

[sandy081]

I do not think we can do anything from VS Code side without such support from Market place.

[davidanthoff]

Where can folks give feedback that this is important? I thought leaving this issue open so that people can still like it might provide that option?

[DanTup]

We're fine with all VS Code Insiders users also getting the Insider extension instead of having a separate insider setting just for our extension.

I don't think it'd be great to tie these together since it'd mean using the insiders version of your extension means that the user needs to be happy to switch also to Insiders Code and with it any other extensions that are publishing Insiders versions. I wouldn't want to go "full unstable" just to do it for one extension (for one, the stability of Insiders extensions could vary wildly across projects).

It also changes the environment somewhat, so you're not getting an accurate test of what happens if you publish the extension (unless ofc you wait for the next stable Code release).

I do not think we can do anything from VS Code side without such support from Market place.

Where can we show support for this? If this case stays closed, I suspect this will be forgotten about but it seems to be a good request desired by several extensions.

[sean-mcmanus]

Yeah, having extension "insider-ness" be independent of VS Code itself is fine with me too. We just need the marketplace to allow us to submit an insider version and then VS Code would need to add an "Install Insider" button next to extensions that also have an insider version. Does the marketplace have some feedback site where this is being tracked?

[eamodio]

They could even just re-use the preview flag -- I would bet users don't know what that is even supposed to mean (I don't -- other than some signal with regard to quality/stability/features -- but very nebulous and I bet doesn't stop bad reviews).

[DanTup]

I guess the Marketplace would still need explicit support for this since people who don't want the preview/insiders/pre-release versions would expect to browser the marketplace and see the release notes (etc.) for the latest stable release, not the preview release.

[eamodio]

Yeah, agreed -- there would definitely be a bunch of plumbing that they'd need to do on the marketplace side.

[sandy081]

@viradhamMS @pkuma-msft We have a popular request to support beta/insiders versions of extensions which in turn needs Market place support.

May I know if there are any plans to have this support in Market place and what is the right Market place forum to raise these requests?

Reopening on users demand :)

[viradhamMS]

Thanks folks. We do have a backlog feature to enable similar requirement. When we get to it, we will reach out to you folks to understand the requirements to finer detail.

For such feedback/questions on Marketplace, Please send a mail to vsmarketplace @ microsoft.com

[sean-mcmanus]

It seems like extensions themselves can implement their own insiders channel without marketplace support for users who have some "yourExtension.insiders" setting set:

1. Downloading some json file that has the current insiders version (and URL) of the extension on activation (we already do this for on the fly a/b testing).
2. If there's a new insider version, the extension can automatically download/install the vsix. And if you want more extension insider usage, you could set the default setting to true for VS Code insiders builds (requiring a package.json rewrite), and users could change the setting to false if they want the release version. This would hopefully solve our current problem in that our manually installed vsix has way too small 0.03% installs compared to the released version. It's on our backlog: https://github.com/Microsoft/vscode-cpptools/issues/1526 .

Theoretically, this mechanism could be implemented in a way that is sharable between extensions via something like:

1. Adding a setting extensions.insiders: [ "ext1": true, "ext2": false].
2. Adding a property to package.json that sets the URL to download that points to a json file with the current insiders version and URL. We actually have 4 additional "offline" installer URLs for different platforms that wouldn't fit into this design, but we could ship just the online installer via this mechanism.

But I don't know how many other extensions would want to take advantage of this.

[eamodio]

@sean-mcmanus that sounds very interesting. Personally I would like for vscode to somewhat "approve" a mechanism like this before I'd employ it in my extensions.

[stkb]

While I was wanting this feature too, I have to say, the improved support for installing from .vsix files (without fear of conflicts) in #39024 is working nicely for me. Most people who have an issue with my extension are only interested in that one thing anyway. So I can attempt a fix, upload a beta .vsix file though github releases, and point them there. If everything's ok, I tell them to stick with that and then with the next marketplace release they're updated automatically and back on stable.

Now of course... it work for all cases; large extensions that want a permanent beta channel. Just wanted to share for anyone that might have missed that change.

[DanTup]

The vsix fix is good; but I've had several beta releases in the last week or so. Telling users to keep updating vsix's kinda sucks. Having auto-updates of betas would make is so much easier - I wouldn't have to convince people to go download another vsix and install it :)

[TylerLeonhardt]

Hey everyone! I work on the PowerShell extension and would LOVE to have a preview/beta channel. In the meantime, I'm probably going to go down the route of having a separate extension similar to what tslint does.

Shoutout to @mattbierner for pointing me here 👍

[sean-mcmanus]

FYI, the 0.19.0 C/C++ extension has implemented a C_Cpp.updateChannel setting with an "Insiders" option that will automatically check our GitHub releases page for a newer Insiders .vsix and download/install that: see https://github.com/Microsoft/vscode-cpptools/tree/release/Extension/src . Users of the Insiders VS Code have that setting defaulted to "Insiders".

[eamodio]

@sean-mcmanus That's very cool -- though I would still really like this to be built-in/provided functionality.

[davidanthoff]

Couldn't the preview tag in a published extension manifest be used for this? If a release has the preview property as true, don't install it, unless a user opts into the "preview" channel for that extension. It seems to me that this wouldn't require any changes on the marketplace but could be implemented in VS Code itself?

[TylerLeonhardt]

To add to that, Ideally we'd want to be able to grab the latest non-preview if we want the stable channel, or the latest preview if we want the preview channel.

[sean-mcmanus]

@davidanthoff "Preview" is currently being used by some extensions for their "release" versions that are not considered "feature complete" enough for 1.0, so you'd be changing the meaning of the "preview" field. It would be better to check the version to see if it contains an "-insiders" which is what our extension uses to implement our "C_Cpp.updateChannel": "Insiders".

[davidanthoff]

I see. In any case, I don’t understand why this can’t just be implemented in VS Code itself, why is there a need for a marketplace update?

[sean-mcmanus]

VS Code's extension API doesn't have a generic concept of "<extension-setting-prefix>.updateChannel", so they would have to add some package.json setting for extensions to provide what setting should be queried for that (or some other equivalent mechanism). If that is done, then they could use an implementation similar to what our C/C++ extension does to bypass the marketplace and query GitHub for new Insider vsix's.

[davidanthoff]

Couldn't VS Code simply store somewhere in its own settings which extension is on a prerelease channel, and then check the marketplace whether there is an extension that has a version number that qualifies it as a prerelease?

[sean-mcmanus]

@davidanthoff How would VS Code determine if an extension is on a prerelease channel? There needs to be some API or package.json for setting that (e.g. an "updateChannel" or "updateChannelSettingName" field in package.json). The marketplace only allows submission/publishing of one version at a time -- we can't submit a 0.20.1 and 0.21.0-insiders without changes to the marketplace implementation (only the last submitted would be used).

[davidanthoff]

But aren't all versions that have ever been published stored on the marketplace? I just tried this with our julia package, and I can get any old version with a link like

https://marketplace.visualstudio.com/_apis/public/gallery/publishers/julialang/vsextensions/language-julia/0.8.0/vspackage

I guess the problem is that the extension can't get a list of all the uploaded extension versions?

[sean-mcmanus]

@davidanthoff Yeah, I think you're right about all the older versions being saved (we used to tell users to use that when there was some breaking regression), but VS Code would need a way to query the marketplace for the list (or the latest Insiders versus non-Insiders). It's possible such an web API exists, but I haven't checked. Our extension queries https://api.github.com/repos/Microsoft/vscode-cpptools/releases for the version list.

[TylerLeonhardt]

That API you linked has whether or not a version is a prerelease or not. Could we go down the list for the last non-prerelease?

Also couldn't help but notice cpptools is following semver so that could also be leveraged, potentially.

[sean-mcmanus]

@TylerLeonhardt Do you mean the GitHub API? That only works for our extension because we post our Insider .vsix's on GitHub and we have custom code that queries it and updates our extension with that. VS Code itself could do something similar to give other extensions the ability to have an Insider channel -- as mentioned previously, there just needs to be some mechanism for exposing the per-extension setting to turn the Insider mode on or off. FYI, our extension has gotten thousands of users on our Insiders, so we consider it a success (compared to tens of users previously, with the "manually discover/download/install the vsix" approach).

[TylerLeonhardt]

Oof my bad. Got confused on who is who in this thread and totally thought the API you linked was from the marketplace 😄

if the marketplace has an API similar to GitHub's... that could be leveraged in vscode to achieve the NuGet Package Manager-like experience of installing extensions - checkbox to allow prerelease/insiders/previews... which filters the list.

[davidanthoff]

This seems an API that can pull all versions that are on the marketplace. I think with that all the pieces should be in place that this issue here could be implemented without any changes to the marketplace:

1. VS Code stores per extension whether it should get pre-release versions for that version or not and exposes a UI for that setting
2. If an extension is configured for pre-release, it pulls the info about all available versions from the marketplace via the API I linked to, and then gets the newest release that has say beta or something like that in the semver.
3. If an extension is not configured for prereleases, it just installs the latest "normal" version, based on the version property.

Essentially I think one could implement this in the same spirit that https://github.com/Microsoft/vscode/issues/12764 was implemented, and distinguish between release and prerelease versions on the version property. CCing @sandy081, given that he did the arbitrary version installation stuff.

[sean-mcmanus]

@davidanthoff I'm not aware of a per-extension setting in VS Code -- maybe the "setting" could just be exposed via the right-click menu in the Extensions pane, like the "Install Another Version..." option is. Also, in order for us to switch to this VS Code implementation, the per-extension setting would need to default to "Insiders" for users of VSCode Insiders, because that's how we get most of our current Insider usage (and enable people who don't want to be on Insiders to change the default).

[TylerLeonhardt]

Implemented a workaround for this by shipping two extensions...

Preview https://marketplace.visualstudio.com/items?itemName=ms-vscode.PowerShell-Preview

Stable https://marketplace.visualstudio.com/items?itemName=ms-vscode.PowerShell

I ended up running into a strange issue trying to release a 2.0.0-preview.1 version of the Preview package:

No support for SemVer 2! :(

So I tried to use 2.0.0.0 instead and npm doesn't support 4 numbers... only SemVer 2 format!

It's been a wild ride... I really hope the code team can do something about this.

Right now extensions can only be X.X.X versioned.

cc @auchenberg who I was chatting a bit about this on Twitter

[DanTup]

In some ways I like that I can't accidentally publish a dev release. I'm really nervous when I type vsce package that I might push a dev build to all users!

I wonder if we could still have a prefix that was vblocked (eg. -dev) even it became possible to upload other pre-release versions (in order to support this). There would need to be some work in VS Code and on the site to let users see/pick between stable/preview releases, but being able to see versions in VS Code now seems like a step towards this.

[beriberikix]

Out of curiosity, where does one file issues against the Marketplace? It sounds like we need to raise awareness to the folks who maintain the Marketplace as well as the vscode team (who are obviously already on this thread.)