Customizable sync list

[renkun-ken]

Thanks for the nice project!

I wonder if it is already supported to customize the list of extensions to sync? I'm working in an offline environment where only allowed extensions should be synced. The whitelist only contains less than 50 extensions.

[assafushy]

+1

[roarbr]

Hi, In the artifacts-directory vscodeoffline will generate a almost empy file called specified.json. Inside this file you may list extensions that you would like to download. This is the content from the file when generated first time:

{
    "extensions": []
}

You may add extensions as a comma separated list like this:

{
    "extensions": [
      "GitHub.vscode-pull-request-github",
      "ms-python.python"
    ]
}

By adding --sync or --syncall flags it will also download 579 recommended extensions in addition to the ones you have listed in specified.json.

I have found one way so far to make the tool only download the extensions listed in specified.json. This are the commands to do this on fedora 34 linux, running sync.py directly, not using docker images:

# Clone git repo
git clone https://github.com/LOLINTERNETZ/vscodeoffline.git
cd vscodeoffline/vscoffline

# Install requirements (this in fedora 34):
pip install --user logzero requests falcon watchdog pytimeparse

# Create empty artifacts dir
mkdir ../artifacts

# Run once to generate specified.json. Do not download code binaries, only download specified extensions,
# not recommended. Nothing is downloaded here, just a one-time-step.
python3 sync.py --artifacts ../artifacts/  --skip-binaries --check-specified-extensions --update-extensions

# Edit ../artifacts/specified.json, add your wanted extensions as described above. Sync again:
python3 sync.py --artifacts ../artifacts/  --skip-binaries --check-specified-extensions --update-extensions

# Result:
$ python3 sync.py --artifacts ../artifacts/  --skip-binaries --check-specified-extensions --update-extensions
[I 210704 21:46:09 sync:534] Syncing VS Code Specified Extensions
[I 210704 21:46:09 sync:340] Adding extension to mirror GitHub.vscode-pull-request-github
[I 210704 21:46:09 sync:340] Adding extension to mirror ms-python.python
[I 210704 21:46:09 sync:539] <VSCExtensionDefinition> GitHub.vscode-pull-request-github (69ddd764-339a-4ecc-97c1-9c4ece58e36d) - Version: 0.27.1
[I 210704 21:46:09 sync:539] <VSCExtensionDefinition> ms-python.python (f1f59ae4-9318-4f3c-a9b5-81b2eaa5f8a5) - Version: 2021.6.944021595
[I 210704 21:46:09 sync:569] Checking and Downloading Updates for 2 Extensions
[I 210704 21:46:09 sync:574] Progress 0/2 (0.0%)
[I 210704 21:46:12 sync:585] Complete

# List downloaded extensions (with dependencies)
$ ls -1  ../artifacts/extensions/
GitHub.vscode-pull-request-github
ms-python.python
ms-python.vscode-pylance
ms-toolsai.jupyter

$ ls -1  ../artifacts/extensions/GitHub.vscode-pull-request-github/0.27.1/
extension.json
Microsoft.VisualStudio.Code.Manifest
Microsoft.VisualStudio.Services.Content.Changelog
Microsoft.VisualStudio.Services.Content.Details
Microsoft.VisualStudio.Services.Content.License
Microsoft.VisualStudio.Services.Icons.Default
Microsoft.VisualStudio.Services.Icons.Small
Microsoft.VisualStudio.Services.VsixManifest
Microsoft.VisualStudio.Services.VSIXPackage

The downloaded files is not a vsix-package, but seems like a file layout for the vscgallery tool.

[roarbr]

Example of the same mirroring from vscsync docker image. Building the image our selves from the code:

# Clone code and build two docker images using docker-compose
git clone https://github.com/LOLINTERNETZ/vscodeoffline.git
cd vscodeoffline
docker-compose build

# List docker images
$ docker images
REPOSITORY               TAG          IMAGE ID       CREATED         SIZE
lolinternet/vscgallery   latest       40309241b687   4 minutes ago   76.5MB
lolinternet/vscsync      latest       74f6e8ee1334   4 minutes ago   51MB

# Assume that you have a $(pwd)/artifactory/ directory with the file specified.json in it. Run the sync docker image with the
# same options to download only requested extensions, not recommended extensions. Run command in docker as current
# user to make sure the files are created with $USER rights.
$ docker run --rm -it -e SYNCARGS="--skip-binaries --check-specified-extensions --update-extensions" -v $(pwd)/artifacts:/artifacts -u $(id -u):$(id -g) lolinternet/vscsync
[I 210704 20:22:17 sync:534] Syncing VS Code Specified Extensions
[I 210704 20:22:18 sync:340] Adding extension to mirror GitHub.vscode-pull-request-github
[I 210704 20:22:18 sync:340] Adding extension to mirror ms-python.python
[I 210704 20:22:18 sync:539] <VSCExtensionDefinition> GitHub.vscode-pull-request-github (69ddd764-339a-4ecc-97c1-9c4ece58e36d) - Version: 0.27.1
[I 210704 20:22:18 sync:539] <VSCExtensionDefinition> ms-python.python (f1f59ae4-9318-4f3c-a9b5-81b2eaa5f8a5) - Version: 2021.6.944021595
[I 210704 20:22:18 sync:569] Checking and Downloading Updates for 2 Extensions
[I 210704 20:22:18 sync:574] Progress 0/2 (0.0%)
[I 210704 20:22:20 sync:585] Complete

$ ls -1 artifacts/extensions/
GitHub.vscode-pull-request-github
ms-python.python
ms-python.vscode-pylance
ms-toolsai.jupyter

[LOLINTERNETZ]

Perfect. As @roarbr suggested, there are a few ways to achieve this behaviour. For the Docker images, setting SYNCARGS is simplest, or if you run the code directly then you can omit various arguments.

The --sync and --syncall` args are really there to make it easier but if you have power-user requirements then it will hopefully cater to your case @renkun-ken.

[roarbr]

A small update.

As I said earlier, the downloaded files is not simple vsix files that can be installed directly into vs code. But the downloaded files contains package name and version and the file Microsoft.VisualStudio.Services.VSIXPackage inside each package is the actual vsix file.

So I tried a command to create the vsix files from the package name + version and copy the Microsoft.VisualStudio.Services.VSIXPackage into -.vsix.

Assuming the working directory is inside vscodeoffline/vscoffline and you have added the two packages to speficied.json as in my previous example and run the sync:

$ ls -1  ../artifacts/extensions/
GitHub.vscode-pull-request-github
ms-python.python
ms-python.vscode-pylance
ms-toolsai.jupyter

Create a new directory to store the vsix-files into. This directory can be copied to offline network for installing into vs code. mkdir code-extensions

Now find each Microsoft.VisualStudio.Services.VSIXPackage and extract the extension name and version. This is one way to do it, other solutions may also work. In this example I copy the files to a new name:

$ for i in $(find ../../artifacts/extensions/ -type f -name Microsoft.VisualStudio.Services.VSIXPackage); do package=$(echo $i|cut -d/ -f5);version=$(echo $i| cut -d/ -f6);cp -auv $i $(echo $package-${version}.vsix);done

# Output from the command:
‘../../artifacts/extensions/GitHub.vscode-pull-request-github/0.30.0/Microsoft.VisualStudio.Services.VSIXPackage’ -> ‘GitHub.vscode-pull-request-github-0.30.0.vsix’
‘../../artifacts/extensions/ms-python.python/2021.9.1230869389/Microsoft.VisualStudio.Services.VSIXPackage’ -> ‘ms-python.python-2021.9.1230869389.vsix’
‘../../artifacts/extensions/ms-toolsai.jupyter/2021.9.1001244851/Microsoft.VisualStudio.Services.VSIXPackage’ -> ‘ms-toolsai.jupyter-2021.9.1001244851.vsix’
‘../../artifacts/extensions/ms-python.vscode-pylance/2021.9.2/Microsoft.VisualStudio.Services.VSIXPackage’ -> ‘ms-python.vscode-pylance-2021.9.2.vsix’

# List files in this directory:
$ ll
total 64208
-rw-rw-r--. 1 x x  17002798 Sep 17 23:26 GitHub.vscode-pull-request-github-0.30.0.vsix
-rw-rw-r--. 1 x x 15746456 Sep 17 23:26 ms-python.python-2021.9.1230869389.vsix
-rw-rw-r--. 1 x x 17176917 Sep 17 23:27 ms-python.vscode-pylance-2021.9.2.vsix
-rw-rw-r--. 1 x x 15810645 Sep 17 23:26 ms-toolsai.jupyter-2021.9.1001244851.vsix

This directory may now be archived and/or copied to another network/computer and the vsix extensions may be installed into vs code using the menu in the IDE, or from the command line (may be scriptet to install several extensions):

# List installed extensions (here the list is empty)
$ code --list-extensions

# Install one extension that I just downloaded with the above solution:
$ code --install-extension GitHub.vscode-pull-request-github-0.30.0.vsix
Installing extensions...
(node:580) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `code --trace-deprecation ...` to show where the warning was created)
Extension 'GitHub.vscode-pull-request-github-0.30.0.vsix' was successfully installed.

# Worked fine. Now list installed extensions:
$ code --list-extensions
GitHub.vscode-pull-request-github

So this make it viable and easy to use vscodeoffline to mirror vs code extensions as vsix packages to install offline or another place without needing to set up vscgallery.

[LOLINTERNETZ]

Thanks @roarbr. It might be worthwhile including this in the Readme (or maybe include a small tool to do that), for people who are just looking to mirror the files as vsix packages, if you think that's useful? We can open a separate ticket. I'm happy to add it when I get a chance but that might not be for awhile.

The file and directory layout used mirrors as much as what Microsoft are doing internally. Vscsync will fetch the package and all of the associated files which are used for various purposes by the gallery. Part of the reasoning here is to keep what is online as consistent as what is offline, not to modify any files, and support gallery functionality. The file structure is also used as a naïve way to manage state and versioning.

Absolutely running the sync process to bulk mirror extensions could be useful in its own right though, without using the gallery. The gallery functionality is likely useful for larger deployments, whereas just exposing the vsix/vsixpackage files might be better for smaller deployments.

[roarbr]

Hi @LOLINTERNETZ. Thanks for the feedbacks.

The gallery seems to be a nice solution where the vs code installations on a internal network is tricked into believing they are connected to the real marketplace. I guess this will work with the install/update mechanism for extensions and so on.

But there are a few steps to install gallery internally, a own web server, dns-redirect and some certificates. The solution to turn the mirror into vsix files are easier to start with and as you say, maybe better for small installations. And easy to script for a small team as the command line options to code can be used to install/uninstall extensions.

If I find time I might create some helper scripts and/or update the Readme and create a pull request, but no guarantees :)