Newer builtin extensions trying to use APIs in proposal in older vscode versions?

[christiandt]

We seem to be experiencing some issues with vscode.git and vscode.ipynb builtins. The extension links to this repo, but please let me know if this is not the correct repo to report these issues.

For VSCode instances with OpenVSX as the configured extension marketplace, several versions of vscode are unable to open jupyter notebooks at all (due to incorrect engine version support in the extension package.json, or unpinned dependencies?). This leads to the extension being installed on older versions of VSCode where the APIs were in preview and not allowed for general consumption.

vscode.ipynb on VSCode version 1.66.0:

Activating extension 'vscode.ipynb' failed: Extension 'vscode.ipynb' CANNOT use API proposal: notebookEditor.
Its package.json#enabledApiProposals-property declares: [] but NOT notebookEditor.
 The missing proposal MUST be added and you must start in extension development mode or use the following command line switch: --enable-proposed-api vscode.ipynb.

vscode.git on VSCode version 1.66.0:

Extension 'vscode.git' CANNOT use API proposal: scmValidation.
Its package.json#enabledApiProposals-property declares: [] but NOT scmValidation.
 The missing proposal MUST be added and you must start in extension development mode or use the following command line switch: --enable-proposed-api vscode.git

Extension configuration in package.json:

"engines": {
    "vscode": "^1.57.0"
  }

Manually disabling automatic updates, removing the newer version, and installing an older version (vscode.ipynb@1.60.0) will make the extension function again.

This issue could be related to https://github.com/microsoft/vscode/issues/149181

[vince-fugnitto]

@christiandt thank you for reporting the issue :+1:

For VSCode instances with OpenVSX as the configured extension marketplace

I see that you've mentioned VS Code, am I correct in assuming the actual application is VS Code or VSCodium? In both cases there should not be a need to consume either vscode.git or vscode.ipynb in your application from Open VSX as they are already included in your application's sources.

The extensions exist on Open VSX for applications that support the VS Code API (such as Eclipse Theia based products), who want to make use of the builtin extensions VS Code provides by default. This is why this repository builds those builtin extensions from VS Code's sources, and publishes them to Open VSX.

[christiandt]

@vince-fugnitto Thank you for the prompt response 😃

Our deployment consists of a few hundred instances of Gitpod.io Openvscode-Server (https://github.com/gitpod-io/openvscode-server), so technically it is VS Code with some reconfiguration/additions. I do agree, there should be no reason for the builtin extensions to be updated from the marketplace, so I was a bit surprised as to why they where being updated automatically.

While they don't show up in the code cli or extension management in VSCode by default, they seem to be re-using the same extension infrastructure, and will show up in the normal extensions interface when manually downgraded to a previous version.

Could there be any chance that the builtin extensions could be managed in the same way as other extensions behind the scenes? If not, could this be a miss-configuration by gitpod.io?

[vince-fugnitto]

Given that you are using openvscode-server there should not be a need to install the builtins or have them updated since they are included by default and are definitely a special case in terms of extensions.

For example, when I quickly checked the public gitpod.io it looks like these builtins cannot be updated, or installed, and can only be disabled.

Could there be any chance that the builtin extensions could be managed in the same way as other extensions behind the scenes? If not, could this be a miss-configuration by gitpod.io?

This is probably a question better suited for Gitpod itself, in the public instance I do not see that builtins are treated as regular extensions.

[christiandt]

There seems to be no safeguarding on builtin extensions that exist on the remote marketplace, which might be some of the cause of this problem.

Here are some simple steps to reproduce this error:

1. Start openvscode-server:

   docker run -it --init -p 3000:3000 gitpod/openvscode-server:1.66.0

2. In vscode install vscode.ipynb extension using the cli:

   code --install-extension vscode.ipynb

3. Refresh browser
4. Try to open any file of filetype .ipynb

Result:

workbench.web.main.js:2642 Activating extension 'vscode.ipynb' failed: Extension 'vscode.ipynb' CANNOT use API proposal: notebookEditor.
Its package.json#enabledApiProposals-property declares: [] but NOT notebookEditor.
 The missing proposal MUST be added and you must start in extension development mode or use the following command line switch: --enable-proposed-api vscode.ipynb.

The error message is different using vscode 1.68.1 (same result as: https://github.com/microsoft/vscode/issues/153100), but the extension will regardless break all .ipynb files.

From the above, it seems possible to install builtin extensions from openvsx, and have them take precedence over the ones shipped with the IDE. Many of the users to which this error has occurred does not know what a cli is, and even less so know of the existence of the code cli, so they have not installed the update using that method.

[vince-fugnitto]

There seems to be no safeguarding on builtin extensions that exist on the remote marketplace, which might be some of the cause of this problem.

Thank you for clarifying :+1:

I see that vscode products don't seem to be robust enough to handle this use-case gracefully, but I'm not sure how critical it is since I do not see a reason as to why users would perform such an operation, especially since these extensions are included by default. If they run into difficulties they should remove these duplicates.

In any case, I do not believe it is an issue with this repository, as this repo's purpose is to build vscode's extensions and publish them to open-vsx for Eclipse Theia based products to use. If you want more graceful handling of such a use-case, it might be worthwhile to open an issue in VS Code directly (ex: you manually built a builtin at a different version and included it to your vscode and something is broken).

[christiandt]

Sorry, the process of reproducing this issue is a lot simpler than initially thought:

1. Start openvscode-server:

   docker run -it --init -p 3000:3000 gitpod/openvscode-server:1.66.0

2. Open browser
3. Wait a minute or two
4. Refresh browser
5. Try to open any file of filetype .ipynb

I do believe you might be incorrect in the way builtins are handled. If you try and start up an instance of vscode connected to openvsx, it will by default update all builtins which have a match on the external marketplace. I am able to reproduce similar behavior on the desktop version of VS Code, but this is not an issue there as long as the microsoft marketplace does not contain the builtins.

Initial state:

State after a few minutes:

[christiandt]

If I read this thread correctly, there are use-cases where by design you would want to override the builtins, but the use-case here in openvsx seems to cause more harm than good: https://github.com/microsoft/vscode/issues/139705

[christiandt]

If nothing else, this seriously diminishes the point of the "Popular" category, as any new installation will start hammering openvsx with updates. Notice the "Reload Required" indicating the installation from openvsx:

At this point it seems really easy to reproduce, but let me know if you want a network dump as additional proof

[vince-fugnitto]

@christiandt I think we are getting ahead of ourselves a bit, what are you suggesting as a fix?

[christiandt]

@vince-fugnitto Can I challenge the need for the builtins to be published to openvsx?

In addition to the builtins being treated in VSCode as any other extension, the major issue is what @sandy081 describes here:

Yes, overriding the builtin extension makes it user extension and they cannot use proposed apis by default. https://github.com/microsoft/vscode/issues/139705#issuecomment-1010389386

So even if we were to lock down engine versions, it would still have this problem to deal with.

[vince-fugnitto]

@vince-fugnitto Can I challenge the need for the builtins to be published to openvsx?

I do not think that would be a feasible solution as it will break all Eclipse Theia based products. It is also unfortunate since we created, and implemented open-vsx.

Yes, overriding the builtin extension makes it user extension and they cannot use proposed apis by default.

Why is there a need to override the builtin extensions, VS Code's UI does not allow it, so for which use-case is it necessary?

[christiandt]

Why is there a need to override the builtin extensions, VS Code's UI does not allow it, so for which use-case is it necessary?

Can you see any changes in the implementation to gitlab.io which would change that behavior? It is clearly possible in the screenshots provided above.

[vince-fugnitto]

Can you see any changes in the implementation to gitlab.io which would change that behavior? It is clearly possible in the screenshots provided above.

I assume you mean gitpod.io and I do not see that from when I tested the actual official instance. I cannot tell what you did locally, it is likely a side-effect of when you installed a builtin manually.

official instance:

[colin-grant-work]

@christiandt, it sounds like the Code OSS instance you're dealing with is automatically updating builtin plugins. If that behavior is not desirable for your use case, it should be disabled for your use case, not by making the builtin plugins unavailable to all users of Open VSX. Have you tried modifying your application configuration to disable extensions.autoUpdate by default?

[christiandt]

@vince-fugnitto let me make it absolutely, definitely clear that neither I, nor any of the users did update or install these extensions manually, and the exact steps needed to reproduce this error is outlined in comment https://github.com/eclipse-theia/vscode-builtin-extensions/issues/102#issuecomment-1171291729. This is not a special use-case, but the default one.

@colin-grant-work Wholeheartedly agree that the solution to this is likely to completely disable automatic updates for all users. I will just have to make sure this is injected into the users directory before it has time to update 😄 There is the broader issue of this causing problems for the broader community, but my perspective would be that this is a strange way to handle builtins from the vscode repo, not something openvsx should have to take into account. (I still feel the popular count will be pretty skewed though 😅)

[vince-fugnitto]

@vince-fugnitto let me make it absolutely, definitely clear that neither I, nor any of the users did update or install these extensions manually, and the exact steps needed to reproduce this error is outlined in comment https://github.com/eclipse-theia/vscode-builtin-extensions/issues/102#issuecomment-1171291729. This is not a special use-case, but the default one.

@christiandt sounds good, as I mentioned before I don't think this is an issue with this repo or the fact the builtins exist, it is likely something you want to bring up to openvscode-server so they can handle it more gracefully.

[christiandt]

@vince-fugnitto Sure, could you clear up the questions I have on the engine version you pin to though? Seems like you could easily set the correct engine version in your extensions, instead of making it as broad as it is? ("vscode": "^1.57.0")

[vince-fugnitto]

@christiandt do you have an example where we do such a pin?

In Eclipse Theia we do not rely on the engines for builtins as they are sometimes not updated by vscode, or reference * so we set the version to the version that corresponds to the vscode we built at the time. If the framework supports an API of 1.53.2 we would not fetch builtins at a higher version than that since it is likely that it will be incompatible. That's why I mention builtins are pretty special use-case for extensions.

[christiandt]

@vince-fugnitto This is the package.json file from https://open-vsx.org/api/vscode/ipynb/1.62.3/file/vscode.ipynb-1.62.3.vsix

{
  "name": "ipynb",
  "displayName": ".ipynb support (built-in)",
  "description": "Provides basic support for opening and reading Jupyter's .ipynb notebook files",
  "publisher": "vscode",
  "version": "1.62.3",
  "license": "SEE LICENSE IN LICENSE-vscode.txt",
  "engines": {
    "vscode": "^1.57.0"
  },
  "enableProposedApi": true,
  "activationEvents": [
    "onNotebook:jupyter-notebook"
  ],
  "extensionKind": [
    "workspace",
    "ui"
  ],
  "main": "./dist/ipynbMain.js",
  "browser": "./dist/browser/ipynbMain.js",
  "capabilities": {
    "virtualWorkspaces": true,
    "untrustedWorkspaces": {
      "supported": true
    }
  },
  "contributes": {
    "languages": [
      {
        "id": "jupyter",
        "aliases": [
          "Jupyter"
        ],
        "extensions": [
          ".ipynb"
        ]
      }
    ],
    "grammars": [
      {
        "language": "jupyter",
        "scopeName": "source.jupyter",
        "path": "./syntaxes/jupyter.tmLanguage.json",
        "embeddedLanguages": {
          "source.json": "json"
        }
      }
    ],
    "notebooks": [
      {
        "type": "jupyter-notebook",
        "displayName": "Jupyter Notebook",
        "selector": [
          {
            "filenamePattern": "*.ipynb"
          }
        ],
        "priority": "default"
      }
    ]
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/eclipse-theia/vscode-builtin-extensions"
  },
  "keywords": [
    "Built-in"
  ],
  "scripts": {}
}

It sounds like there might be a chance you could possibly pin it to a specific version then, instead of the broad pinning?

[vince-fugnitto]

It sounds like there might be a chance you could possibly pin it to a specific version then, instead of the broad pinning?

The engine comes from vscode as I mentioned, and they generally do not update often:

https://github.com/microsoft/vscode/blob/1de164e8ff00d9ef3198a8f029e65c8a3533b101/extensions/ipynb/package.json#L8-L10

It is why we rely on the version since we can guarantee that this extension was built against a specific version of vscode.

[christiandt]

Ok, makes sense. Thanks for clarifying 👍

[christiandt]

and they generally do not update often

Are you sure about that though? https://code.visualstudio.com/api/working-with-extensions/publishing-extension#visual-studio-code-compatibility

we can guarantee that this extension was built against a specific version of vscode

Wouldn't you need to, for the API support?

[christiandt]

I realise that setting the version to be equal to engine might be a bit too strict, but wouldn't you be able to do for instance version: 1.62.3, engine: 1.62.x at least? Or is even greater flexibility needed?

(I still do see this as a problem caused by microsoft, and not by you guys, even more so with the never changing engine versions as you say 😓 )

[vince-fugnitto]

Are you sure about that though? https://code.visualstudio.com/api/working-with-extensions/publishing-extension#visual-studio-code-compatibility

I'm quite sure, what you linked are external extensions that need to specify an engines for API reasons. However, given that the builtins are part of the sources they are generally not updated. Here are a few examples (note: vscode is at version 1.68):

Just to name a few...

• https://github.com/microsoft/vscode/blob/1de164e8ff00d9ef3198a8f029e65c8a3533b101/extensions/html-language-features/package.json#L10
• https://github.com/microsoft/vscode/blob/1de164e8ff00d9ef3198a8f029e65c8a3533b101/extensions/bat/package.json#L9
• https://github.com/microsoft/vscode/blob/1de164e8ff00d9ef3198a8f029e65c8a3533b101/extensions/git/package.json#L9

As I mentioned multiple times already the builtins are special use-cases, they should not be consumed outside of Theia-based applications since there is no real reason to, the builtins are already included in your application. The way we handle the builtins is through version since we guarantee that the extension was built at that specific version of vscode and have logic to handle it.

I don't think there is much else to discuss.

[christiandt]

Agreed, it is hard to blame anyone for this scenario, but on the bottom of that list would be you guys 😄

please let me know if this is not the correct repo to report these issues

I will answer myself and anyone else reading this thread; this is not the correct repo to report these issues. Anyone annoyed that any incompatible versions are installed (based on engine version) on other versions of VSCode should rather take it up with the sources (examples linked in comment above).