`streetsidesoftware.code-spell-checker` missing in VSX but present on website

[Scandiravian]

I updated my flake sources today and got the following error:

error: attribute 'code-spell-checker' missing

       at /nix/store/pdwfrbvzvya04vwx1712fdqqpbnqfbdd-source/home-manager/common/vscode/extensions.nix:39:5:

           38|     shd101wyy.markdown-preview-enhanced
           39|     streetsidesoftware.code-spell-checker
             |     ^
           40|     tamasfe.even-better-toml

I looked through the commit history and noticed that the extension has been removed in the update 12 hours ago (https://github.com/nix-community/nix-vscode-extensions/commit/fd8cee8a7fecae85c163938237c8db1cf219947e#diff-db06c147305f13309bc4ca7b66da962d0b9e166ae2ed780c000531d2d78841ebL1974)

I considered that it might have been removed from VSX, but looking at the site, it seems to still be available https://open-vsx.org/extension/streetsidesoftware/code-spell-checker (though the site version is 2.17.1, while the one in the repo was 2.16.0)

Is there a reason for the extension disappearing or could this be caused by a bug in the automatic generation of the extension itself?

[Scandiravian]

Just saw https://github.com/AmeerTaweel/nix-vscode-marketplace/issues/3 in the upstream repository. It's a possibility that the extension dropped out of the top 10k, though with 155K downloads I'm not sure that is actually the case

[deemp]

Hi! Both Open VSX and VS Marketpace provide a pretty unreliable info about their sets of extensions. That's why, one day, an extension may be removed from a list and appear there the next day. E.g., the extension you mentioned is now present here. https://github.com/nix-community/nix-vscode-extensions/blob/85df189216b0c9efb27ec803c1080d54086303c2/data/cache/open-vsx.json#L1978

[Scandiravian]

Hi! Both Open VSX and VS Marketpace provide a pretty unreliable info about their sets of extensions. That's why, one day, an extension may be removed from a list and appear there the next day. E.g., the extension you mentioned is now present here.

https://github.com/nix-community/nix-vscode-extensions/blob/85df189216b0c9efb27ec803c1080d54086303c2/data/cache/open-vsx.json#L1978

Hi @deemp,

Thanks for the explanation!

I was wondering if it would be a solution to grab all of the plugins in vsx from their sitemap (mentioned in this issue comment) to get a complete list of all the extensions? I think that would avoid this issue, as the sitemap should be deterministic. The sitemap also has a "lastMod" field, which could be used to determine which plugins it's necessary to update

Alternatively, would it be a solution to run a diff between the old and new data to see if any entries are missing? If they are, they could be checked individually against the API to make sure they're only removed if they're no longer present in the repository

I'll be happy to try to do a POC of either of these solutions, but unfortunately I'm not familiar enough with Haskell to be able to modify the current solution :disappointed:

If my questions are not clear, let me know and I'll try to rephrase them

[deemp]

Hi @Scandiravian!

I was wondering if it would be a solution to grab all of the plugins in vsx from their sitemap (mentioned in https://github.com/eclipse/openvsx/issues/649#issuecomment-1378314413) to get a complete list of all the extensions?

The counts of extensions obtained via json API and via the endpoint are the same.

• https://open-vsx.org/sitemap.xml
• https://open-vsx.org/api/-/search?includeAllVersions=false But the API that we use provides much more info.

Alternatively, would it be a solution to run a diff between the old and new data to see if any entries are missing?

That's a good point. I described a possible solution in https://github.com/nix-community/nix-vscode-extensions/issues/17

[Scandiravian]

Thanks for the response @deemp

TL:DR: Closing this in favour of https://github.com/nix-community/nix-vscode-extensions/issues/17

The counts of extensions obtained via json API and via the endpoint are the same.

As mentioned I'm not strong in Haskell, so re-reading the VSX logic I can see, that the extension only performs a single call to VSX, where my previous belief was that it performed multiple calls and used paging. With this in mind, I can see that my proposed solution doesn't bring any benefits compared to the existing implementation.

Thanks for making me aware of my misconception :blush:

That's a good point. I described a possible solution in https://github.com/nix-community/nix-vscode-extensions/issues/17

That seems like a better approach than what I had in mind. I'll close this issue in favour of yours, since your issue ticket describes a more general approach to fixing the issue instead of relating to a single missing extension

Hopefully the funding and governance of VSX will find a good resolution, so it can continue to be part of this excellent repository (blog post on potential shutdown of VSX)