search

microsoft / vscode

Visual Studio Code
https://code.visualstudio.com
MIT License
163.39k stars 28.94k forks source link

Allow extensions to publish beta releases and users to opt-in to them #15756

Closed DanTup closed 2 years ago

DanTup commented 7 years ago

Apologies if this has come up before; I'm unable to find any related issues.

It would be cool to be able to publish extensions in a beta form (similar to Code Insiders) so that they can be run "in production" with real users that have opted-in. This would reduce the disk in publishing an extension because you can push it to a smaller number of users for testing (I probably don't need to sell this to you, I guess you have Insiders for very similar reasons).

I know we can package and distribute extensions ourselves for now, but it's not as easy (eg. updating is not automatic).

I originally thought the "Preview" flag might've provided this functionality but unless I've overlooked something, it does not seem to.

viradhamMS commented 5 years ago

@beriberikix - You can file issues here itself, we work very closely with VSCode group and it will reach us pretty quickly (or) feel free to directly post to vsmarketplace at microsoft dot com

natanfudge commented 5 years ago

A feature like this would help out immensely for pushing new features. I'm hoping the marketplace/vscode team will get on it soon!

sandy081 commented 5 years ago

Currently most of the extensions are following a separate extension for extension insiders/nightly. This way, users can opt in to install either stable/insiders version of extension and disable the other. It is also NOT limited to insiders/stable.

Any feedback on this solution?

DanTup commented 5 years ago

I don't think it's the best solution - it means you're listed twice in the marketplace, have two sets of reviews/stats, and have to fiddle around when publishing to change the extension ID.

It also means extensions that reference each other by ID (for example Dart & Flutter are dependent and call exported APIs) need some faffing around too (to know to check which version is installed).

And of course, we'd need to do our own checks that the user didn't install multiple versions.

I think being able to publish pre-release versions to the marketplace (even if it's a white-listed set, like -beta) and let users have a flag per-extension to opt in the using pre-release versions would be far better (and I don't think it should be tied to VS Code Insiders).

sandy081 commented 5 years ago

Your concerns are valid.

Another approaches we are discussing which is based on what is suggested here

Approach 2: Defining Insiders Flag (VS Code hosted solution)

Drawbacks

Approach 3: Insiders version tag (Marketplace hosted solution)

I would prefer Approach 3 which is correct IMO and has no drawbacks compared to other solutions. But this has a dependency on Marketplace to support this feature.

davidanthoff commented 5 years ago

Yes, I'm also much in favor of approach 3!

I think it would be nice if the pre-release tags would support things like v1.4.3-alpha.2, v1.4.3-beta.3, v1.4.3-rc.1 etc. and then would automatically do the right thing. If that is too complicated, maybe just v1.4.3-insiders.3 would do the trick as well. But it would be good if one could publish consecutive pre-release versions that are uniquely identified.

DanTup commented 5 years ago

I also prefer 3, since it seems weird if we have to keep revving non-preview version numbers for preview/insiders builds (however if it's likely to take significantly longer because it needs marketplace changes, I'd revise my preference ;-))

I'd also ask that -insiders builds can have additional build numbers (eg. -insiders.1, -insiders.2, -insiders.3 etc.).

eamodio commented 5 years ago

I also strongly prefer 3

sean-mcmanus commented 5 years ago

As mentioned previously, our C/C++ extension implements this ourselves with our C_Cpp.updateChannel settings, so if the marketplace implements something similar it is important that our extension is able to:

  1. Default to Insiders for VS Code Insiders.
  2. Be able to show a prompt such as "Insiders version is available. Would you like to switch to the Insiders channel?: [Yes][Ask Me Later][Don't Show Again]" that can change the version.

Without those 2 features our Insider usage would be too low to be useful. Our code that does this is open source, but it's not packaged to be re-usable by other extensions.

As mentioned by the other users, some sort of 0.23.0-insiders, 0.23.0-insiders2 mechanism is desired: see https://github.com/microsoft/vscode-cpptools/releases .

The main disadvantage with our method is that our Insider versions are not available via the "Install Another Version..." UI in VS Code...so if an API was available to make that fixable I'm not sure why we'd benefit from a marketplace change.

TylerLeonhardt commented 5 years ago

Without knowing the true architecture of the VSCode marketplace, I can only guess... But since vsix's are very similar to nuget packages, my guess is that the marketplace (VSCode, VisualStudio, etc) is some fork of NuGet.org (or NuGet.org is some fork of the marketplace). Also, the PowerShell Gallery is a fork of NuGet.org.

Both NuGet.org and PowerShell took approach 2. My guess is because of the risk of doing approach 3 and making sure that we are still backcompat. Imagine trying to replace a bunch of SemVer v1 code with SemVer v2... There could be a ton of side effects.

With all that said, I would prefer approach 3 because it's the technically correct approach... But my understanding is that that's a different team and this would be a large ask so it's tough on that front.

Im voting for Approach 2 only because I think a lot of Approach 2 is needed for Approach 3 (like the toggle in vscode to show/hide insiders) and it will take the least amount of time and coordination.

All the while, the marketplace team could be working on Approach 3... And then vscode uses that version instead of the preview flag when their work is done.

sandy081 commented 5 years ago

I think with Approach 2 extension authors will have troubles in version their stable and insiders versions. I really do not know what the model would be for versioning stable and insiders with major.minor.patch. For example if the current stable is 1.1.0 what is the next insiders version?

DanTup commented 5 years ago

If you went with 2, I'd probably just use even minor versions for stable releases and odds for insiders to keep things in order:

1.0.0 <-- stable
1.1.1 <-- insiders version of 1.2
1.1.2 <-- insiders version of 1.2
1.2.0 <-- stable
1.3.1 <-- insiders
1.4.0 <-- stable

It's not perfect since you can't have insiders versions of patch releases, but that doesn't seem like a huge issue IMO.

(I'm not saying I'd like 2, but if the difference between delivering that and 3 is a long time - it's been over 2 years since I opened this already - then I'd take it!)

TylerLeonhardt commented 5 years ago

I agree with @DanTup. That seems reasonable if it means we get the feature a lot sooner.

With the PowerShell extension, we recently followed the Python extension and switched to date-based versioning like 2019.8.0 where it's year.month.buildThatMonth. Leveraging that versioning scheme, with the preview property in the package.json determing prerelease (aka Approach 2) would work nicely for us.

P.S. We switched to that versioning scheme because we accidently shipped the preview extension (2.0.0) to the stable extension... It was a very painful experience to mitigate. We couldn't risk using Semver anymore having 2 extensions to manage.

davidanthoff commented 5 years ago

If we went with 3), one could have a concept of a "channel': one could have a nightly channel, a beta channel (that is more stable and has fewer releases) etc.

The versioning scheme could be v1.20.3-nightly.4 and v1.20.3-beta.2 etc.

At the end of the day, I think that is what I'd really like to have and it seems to be really useful in lots of other contexts (Windows, Edge etc.)

janosh commented 5 years ago

This would a great feature addition! Any timeframe on when this might land?

vchuravy commented 4 years ago

I seem to be getting different versions depending on whether I am on VSCode Insiders

Name: C/C++
Id: ms-vscode.cpptools
Description: C/C++ IntelliSense, debugging, and code browsing.
Version: 0.26.2-insiders2
Publisher: Microsoft
VS Marketplace Link: https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools
Name: C/C++
Id: ms-vscode.cpptools
Description: C/C++ IntelliSense, debugging, and code browsing.
Version: 0.26.1
Publisher: Microsoft
VS Marketplace Link: https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools

But it looks like cpptools manually implemented that feature... https://github.com/microsoft/vscode-cpptools/pull/3221

sean-mcmanus commented 4 years ago

@vchuravy Yes, and that's the default behavior, but it can be overridden via our C_Cpp.updateChannel setting. There's a bug preventing it from working for remote machines though (https://github.com/microsoft/vscode-cpptools/issues/3833).

vchuravy commented 4 years ago

It would be fantastic if extensions wouldn't have to implement this feature, but VSCode could offer it by default.

sean-mcmanus commented 4 years ago

@vchuravy Yeah, I agree. Our implementation is open source if any other extension wants to do something similar, but we haven't packaged it into an easily re-usable npm module. Other than the remote bug, our insider system has worked really good for our extension, allowing us to find/fix important bugs before they get released to a wider audience (and allowing users to get updates earlier than our slower 1.5 month non-Insiders release cadence).

davidanthoff commented 4 years ago

Is there any hope that this will make it to the top of the list of the core dev team? I think given that this needs integration with the marketplace, it really can only be done by the core team.

Every time we release a new version of our extension, I wish we had this. In my mind this is single feature that has the most amazing potential to increase the quality of the entire extension ecosystem.

GustavoASC commented 4 years ago

I would love this

davidanthoff commented 4 years ago

We now caved and are publishing a second version of the Julia extension that is an insider build. The whole situation is really not great because the normal and the insider extension now conflict with each other, but it is better than nothing.

If anyone is interested in the Azure pipeline we use that creates the insider build, take a look here. It makes the necessary modifications to the package.json and swaps out the README.

I still really hope that at some point we'll get a proper solution for this...

DanTup commented 4 years ago

From reddit: https://www.reddit.com/r/vscode/comments/fakre5/hey_extension_authors/fj0r2az/

I can speak to beta extensions. This is one of the most requested features for the marketplace and we're actively working on this one.

moshfeu commented 4 years ago

Having 2 extensions - the regular and the insider one raise a problem so the user can't have both of them. I've not digging into it but probably the issues are with duplicate ids of commands, views etc. In principal, we can run a task to replace the ids with insider ids but it's very uncomfortable.

J-Fields commented 4 years ago

@moshfeu I'm eagerly awaiting beta releases as a first-class citizen, but as a workaround for your issue, you should be able to have the regular version look for the insider version (with this api) and disable itself if the latter is installed.

moshfeu commented 4 years ago

Thanks @J-Fields . The conflicts coming from the "contributors" - for example, buttons and commands. I'm not sure how using the API can solves this 🤔

misak113 commented 3 years ago

This would prevent a lot of headaches to extension users :) I'd prefer the possibility for the user to easily rollback to the latest stable version if he accidentally opted-in ;)

anburocky3 commented 3 years ago

It would be cool like Google Play Store So the entire journey from alpha to production would be covered here.

isidorn commented 3 years ago

Upstream https://github.com/microsoft/vsmarketplace/issues/165

isidorn commented 3 years ago

@sandy081 and me have been thinking about this topic and we understand that there are a lot of problems with the current approach of publishing two extensions: confusing for users, inconsistent across extensions, painful to manage both. We believe that the current separation of insiders and regular into two extensions does not make sense. Here's our proposed solution:

Drawbacks of this approach

Above drawbacks will be lifted once MP supports prerelease version tags.

Please let us know what you think about this proposal. We are looking forward to the feedback. Thanks 👏

Authors of extensions that have nightly versions @eamodio @connor4312 @alexr00 @aeschli @brettcannon @davidanthoff @dantup @wardengnaw @polinasok @hediet @greazer @TylerLeonhardt

hediet commented 3 years ago

@isidern I like the proposed solution!

Would vscode-stable users who installed an insider-extension also be updated to stable-builds of that extension if the stable build is newer than the insider build? (I think this makes sense if the extension does not always have an insiders build)

In this case, insider-users who got updated to stable should still get the newest insider build of the extension once it gets published. Thus I think the decision what to update to should not be based on the stability of the installed extension, but rather be a separate (user-configurable) flag.

sandy081 commented 3 years ago

@hediet

Thats a good point to note and Yes, we will be doing that - Irrespective of being on insiders/stable version of an extension users we will always have the best latest ie.,

chrmarti commented 3 years ago

@rchiodo Mentioned using the "engines" property to limit a release to VS Code Insiders, not sure everyone is aware of that option:

    "engines": {
        "vscode": "1.59.0-insider"
    },
hediet commented 3 years ago

@rchiodo Mentioned using the "engines" property to limit a release to VS Code Insiders, not sure everyone is aware of that option:

    "engines": {
        "vscode": "1.59.0-insider"
    },

But 1.60.0 is newer than 1.59.0-insider, right? At least by the semver spec.

isidorn commented 3 years ago

Updated the proposal based on feedback (added drawbacks) and simplified that Stable Extensions would get downloaded for any VS Code flavour and users would have to opt into getting insiders extensions. We can later change this that by default VS Code insiders gets Insiders extensions. This is out of scope for this proposal.

brettcannon commented 3 years ago

What is the plan for user discoverability when it comes to this? With a separate nightly extension you get the SEO for your insiders version by showing up as a separate search result. Bundling under the same extension ID means you lose that. So would users then be expected to check every extension to see if there's an insiders version they can switch to? Or would extensions come up with their own way to "advertise" their insiders version to users and ask them to click the right thing to switch? Or would there be an execCommand we could use as part of a notification prompt to help users flip over to insiders?

jakebailey commented 3 years ago

The proposed plan would work well for Pylance; I designed our versioning to follow semver (technically, calver) with this eventuality in mind. Our stable releases are like 2021.9.1, and our insiders are like 2021.9.2-pre.1 (insiders builds are always semantically newer than stable builds).

I was worried about the case where insiders users update to stable builds, when we want them to move up to other insiders builds, but I this already happens now as VS Code can update the extension to stable before we get a chance to manually install the VSIX. When a stable build of Pylance is release, we release a corresponding insiders build, so it's probably rare that anyone checks for an update during that small period and would work better than our manual daily checks.

rchiodo commented 3 years ago

I was hoping the default for VS code insiders would be to install the latest insiders (or stable if newer build)

Then it would happen automatically. The special ask the user if they want to opt in means nobody will end up using our insiders bits.

Right now we're using the engines property to accomplish just that. All VS code insiders users get our insiders version automatically (or stable version if it's newer).

sean-mcmanus commented 3 years ago

@isidorn VS Code Insiders defaulting to the latest Insiders extension (or stable if it's newer) is also a "requirement" for our extension to switch to this mechanism (instead of our current custom one) -- otherwise, Insider usage would be "too low" (we could probably provide telemetry data on this if you want).

jakebailey commented 3 years ago

Similar for Pylance; our custom insiders system also respects the Python extension's insiders setting, which gets us a more reasonable number of users. It's hard to make this discoverable.

connor4312 commented 3 years ago

I think we should not talk about "newer" or "latest" builds, but higher versioned builds.

Irrespective of being on insiders/stable version of an extension users we will always have the best latest ie., if stable is latest than insiders then always stable.

This is not a good behavior in my opinion. For example, often in recovery releases I'll go back and publish, for example, a 1.60.2 release even when the 1.61.0-insiders.1234 is available. In this case users on 1.60.1 should be updated, but those on the 1.61.0 nightly should not be. That is:

Aside from that I think the proposal looks good.

sean-mcmanus commented 3 years ago

@connor4312 I believe all references to "newer" and "latest" do mean "higher versioned" builds, and the release of 1.60.2 would not affect the 1.61.0-insiders if it's released later, i.e. it's just a word choice difference.

connor4312 commented 3 years ago

Yea, just want to clarify that there should not be any chronological implication when we talk about version numbers. I thought there was currently a bug in the marketplace where that was assumed, but it looks like it's actually a separate issue (extension auto update only checks for the highest versioned release of an extension, but if its engines.vscode is not compatible with the current vs code version the auto update stops working; I misremembered and thought the problem was that it only returned the most recently published version)

heartacker commented 3 years ago

if insider avaible, IMO, I want this: before installed: image

after installed: image

DonJayamanne commented 3 years ago

I think we should not talk about "newer" or "latest" builds, but higher versioned builds.

I'd like to add, the current behavior works very well for the Jupyter extension. We have releases for both stable and insider builds, and we ship recovery builds for stable every now and then, and insider builds on a daily basis

For Jupyter extension, if used wants insider builds, we just ask then to use the vscode insider builds (ie all vscode insider users automatically get the insider build of Jupyter extension). We haven't had any request from users to have insider builds of the extension in the stable version of vscode.

DanTup commented 3 years ago

If a version has been published as insiders, it can not be promoted to stable, a new version has to be pushed. Can create strange version history for extensions

To clarify - does this mean version numbers? Eg. If I publish a 1.25.0 with --insiders, I cannot then also publish 1.25.0 without --insiders? Or does it just mean that we have to push the file again and can't just flip a flag? I'm assuming it's likely the former if the Marketplace isn't handling this and VS Code is instead tracking which versions were marked as insiders? (out of curiosity - how is this working is vsce just adding some additional metadata to the vsix?).

I think having the marketplace show the Insiders version number might be confusing, but it sounds like that's unavoidable without the Marketplace being updated. I'd certainly live with it to have Insiders versions :-)

I agree with other users though - having Insiders users automatically get Insiders by default would be nice to improve usage (or at least make the install view have the Insiders option more prominent).

sandy081 commented 3 years ago

@brettcannon Your discoverability concerns are valid and we will definitely address this while coming up with UX mock ups. Our current thinking is to let VS Code handle this and not by extensions, so that users will get unified consistent experience.

@connor4312 Just to be clear about current proposal - We are not changing current version semantics ie., extension still supports only major.minor.patch. We defer supporting prerelease version tag until Marketplace supports it. Your scenario of publishing 1.60.1 when there is 1.61.0-insiders is supported only then. With the current proposal you still have to follow existing major.minor.patch version semantics but you can tell VSCE / VSCode that a specific version is a an insiders/prerelease one. Unfortunately you have to release a newer/higher version for your insiders, if you have a stable release. So in your case, you have to publish 1.60.1 as stable and 1.60.2 as insiders. This is mentioned as a drawback to the proposal and I agree that it does not sound good and adds some strange version gaps but this temporary and will be lifted once MP supports prerelease tags.

To clarify - does this mean version numbers? Eg. If I publish a 1.25.0 with --insiders, I cannot then also publish 1.25.0 without --insiders? Or does it just mean that we have to push the file again and can't just flip a flag? I'm assuming it's likely the former if the Marketplace isn't handling this and VS Code is instead tracking which versions were marked as insiders? (out of curiosity - how is this working is vsce just adding some additional metadata to the vsix?).

@DanTup your assumption is correct. Yes, with current approach VSCE/VSCode are just tracking the versions marked as insiders. Once MP supports prerelease versions we will move to it. Yes, we will be adding this as metadata to vsix for now.

brettcannon commented 3 years ago

What would the MP link to for downloading? The Python extension gets sideloaded a lot due to aggressive firewalls blocking VS Code hitting the MP for updates, so having the download link in the MP be the stable release would be good. Making the link just the highest versioned thing, insiders or not, can cause issues for extensions like Jupyter where users don't realize that the thing they are downloading from the MP is actually the insiders build and not the stable build.

DanTup commented 3 years ago

@brettcannon my understanding is that there will be no changes to the marketplace in the short-term for the proposal above, so it will just show the highest version number regardless of stable/insiders (because it won't know which are stable and which are insiders - only vsce and VS Code will understand that).

It's definitely not perfect, but if it's really likely to be 18 months for the Marketplace to get support, I'd rather have this. It's been almost 5 years since I opened this issue 🙃

eamodio commented 3 years ago

Before I created a separate Insiders extension for GitLens (and the same for GitHub Repositories) I think I would have (for the most part) loved this proposal, but now I feel there are significant issues with using the same extension id for both a stable and unstable build, at least in how VS Code handles data for them.

Having shared data storage for stable and unstable extensions, can be hugely problematic, not only for backward compatibility, but now you will have to deal with users switching back and forth between versions -- and with synced data this can get even more problematic. Granted extensions internally could try to deal with this, by storing their own data in separate stable vs unstable buckets, but that adds even more complexity (and must be understood before you hit the problem).

VS Code could, of course, store stable vs unstable data in separate buckets automatically and that might address most of my concerns.

Outside of the data concerns, I also feel like versioning will be challenging -- not only just getting the right thing published, but also for error reporting (since the main parts of the version are going to be either the same of very close). I definitely like how today both GitLens & GitHub Repositories use standard versioning 1.5.0 while the insiders version are YYYY.MM.DDHHMM. Which make it easy to tell the difference, but also how recent your insiders build it (since they are basically nightly builds). Maybe the --insiders or maybe better yet a --prerelease-channel <name> flag could not only specify the channel, but also affect the versioning structure -- so maybe all prereleases are versioned by the date.

Also if we think of this concept all as pre-release "channels" where the user can choose the channel they want to be in, it will be something easier to manage for authors, users, and the marketplace. On the marketplace, you could then show version histories by channel, download/upgrade by channel, etc. And maybe the number of channels and names are publisher controlled -- or maybe they are something we try to standardize on. And maybe vscode insiders get, by default, opt-ed into a pre-release channel, which they can opt-out of individually or globally. But in this way we aren't just thinking about version numbers -- version numbers are a secondary consideration after the channel.

Anyway that my 2c.