Convert Dependencies to be Installed into Lib Folder

[wbond]

Since Sublime Text 3.0, we've shipped a Data/Lib/ folder that contains Python-version-sepcific folders that are automatically added to the sys.path. The intention of adding these was to set the environment up so that Package Control would switch to using a more sane way to install dependencies.

Currently, PC puts dependencies into a package folder with the name of the dependency. However, this was messy, and due to the load order of packages in Sublime Text, a special packages was created to inject the dependencies in the environment before any real packages were loaded.

The work on this has started, and https://github.com/wbond/package_control/blob/four-point-oh/2_bootstrap.py works on migrating dependencies in 0_package_control_loader to the Lib/python33/ folder.

The approach to installing dependencies moving forward will be:

1. [ ] Finish dropping the concept of 0_package_control_loader. Thus all code related to this will be ripped out.
2. [ ] The code that installs dependencies needs to be updated to installed into the Lib/ folder for the appropriate Python version. At a high level, the installation should be done in such a way that it will look like a .whl was installed. The reason for this is that we'd like to support the ability to install .whl files in the future.

There will end up being quite a bit of code in https://github.com/wbond/package_control/blob/four-point-oh/package_control/package_manager.py modified to deal with the new approach to dependencies.

The concept of load_order for dependencies can be dropped since the Python import functionality will deal with what we used to handle via 0_package_control_loader.

[deathaxe]

It means both, the module and the dist-info path should exist in Lib then?

• lxml
• lxml-4.6.1.dist-info

[wbond]

Yes. There is code to do that already for the migration of existing dependencies. See https://github.com/wbond/package_control/blob/four-point-oh/package_control/wheel.py.

[wbond]

Also https://github.com/wbond/package_control/blob/four-point-oh/2_bootstrap.py.

[deathaxe]

Just found it some minutes later.

Hmm, well. Introducing the most non-pythonic part of python in ST.

While it opens the door to install packages via pip, I always disliked the way how normal libraries and such meta folders are cluttered next to each other, especially if meta file folders have different names than the package itself.

``` backrefs backrefs-1.9.0.dist-info bracex bracex-2.0.0.dist-info coverage coverage-1.0.0.dist-info dateutil dateutil-2.8.1.dist-info jinja2 lsp_utils lsp_utils-1.11.2.dist-info markdown markupsafe markupsafe-1.1.1.dist-info mdpopups mdpopups-3.7.5.dist-info package_control.py package_setting_context-1.0.2.dist-info package_setting_context.py pathlib-1.0.1.dist-info pathlib.py pygments pygments-2.0.2.dist-info pymdownx pymdownx-6.2.1.dist-info pyte pyte-0.8.0.dist-info python-jinja2-2.10.1.dist-info python-markdown-3.1.1.dist-info python-toml-1.0.0.dist-info pywinpty-0.5.7.dist-info pyyaml-5.1.1.dist-info requests requests-2.15.1.dist-info ruamel ruamel-yaml-1.2.0.dist-info six.py sublime_lib sublime_lib-1.4.0.dist-info tests.py toml wcmatch wcmatch-1.1.1.dist-info wcwidth wcwidth-0.1.7.dist-info winpty yaml yamlmacros yaml_macros_engine-2.1.0.dist-info __init__.py ```

[shardulbee]

@wbond I'm currently looking at tackling this. Could you help me understand why the concept of load_order was introduced?

[wbond]

@shardulbee Just be aware that https://github.com/wbond/package_control/issues/1516 really needs to happen before this.

The load_order was introduced because I didn't write a Python loader for dependencies. Thus we needed to make sure all dependencies of a dependency were added to sys.path by the time they were needed.

[shardulbee]

Can't we assume right now that all dependencies should be installed to the python33 folder, and then update to install to python38 once the schema is updated?

[wbond]

I'm not going to do a release with part of the work, and I'd much rather the code for installing dependencies make sure it has the info necessary to know where to install it. If we don't take into account the fact that we need to get version info from the package and then filter dependencies to find one that is compatible, we'll probably end up throwing away a bunch of code.

Packages will indicate what plugin_host they want to use via a file named .python-version in the root of the package. We'll need to extract this data from the package, and then make sure downloads exist for the correct version of Python.

On a related note, the dependency installer code needs to be updated to handle a few things:

1. The old folder structure, as described at https://packagecontrol.io/docs/dependencies. For the new schema, using this old folder structure will be indicated by a key, something like "legacy_folder_structure". Such releases won't work for linux_arm64, nor Python 3.8 dependencies.
2. All other releases should be the dependency files in the root of the zip file.
3. Support for .whl files, which are zip files with some structure

[deathaxe]

I wonder whether dependecies should be handled in a dedicated manager module and/or class rather than enlarging PackageManager class any further especially as the way they need to be handled changes a lot compared to normal packages.

[rwols]

Once dependencies are migrated, PC still starts downloading "missing dependencies" into the $packages folder. I'm not sure if that's intended?

[deathaxe]

This part is not yet implemented. That's the next step.

[rchl]

I have a related question/concern.

Dependencies can include settings file (among other files) in the root folder. For example mine and mdpopups do.

Has it been considered how to handle that? Would only the st3 part be moved into the Lib folder?

[wbond]

@rchl Unfortunately you'll have to find a different method to do what you were doing before. Dependencies were never really intended to do that.

[rchl]

At this point, it doesn't matter much if it was intended or not because it was possible and it has happened. :)

I don't know how many dependencies are affected by it but mdpopups is one that is rather widely used at least. I'm not trying to be difficult or demand anything but it could still be considered to handle that somehow.

[deathaxe]

mdpopups needs some changes to be compatible later on. Dependencies are meant to be libraries only without plugin-like behaviour. Those not have been respected it, will break.

[wbond]

At this point, it doesn't matter much if it was intended or not because it was possible and it has happened. :)

Unfortunately sometimes things have to break to move forward. There is no way to preserve dependencies with package files moving forward with multiple plugins hosts.

Plugins using mdpopups will just need to bundle whatever resources mdpopups provided before. There will be a release of PC4 and if mdpopups hasn’t migrated at that point, plugins will break.

There will be some time before PC4 is released to fix this, but packages can start moving away from putting package files in dependencies now.

[wbond]

If someone wants to highlight how package files are being used in mdpopups, I’m sure we can suggest an alternative approach.

[deathaxe]

mdpopups (which the only known dependency with package level resources next to lsp_utils) uses sublime-settings files and default.css as a "package level resource".

The default.css can easily be moved to the library.

If a dependency uses sane hard coded defaults for settings, the only effort is to provide an alternatative to "edit_settings" command, which pulls the default settings file from another resource.

Both of those breaking changes are fixable.

[rchl]

If a dependency uses sane hard coded defaults for settings, the only effort is to provide an alternatative to "edit_settings" command, which pulls the default settings file from another resource.

It wouldn't be possible for a dependency to provide a command in any way though since none of the files that can provide those (.sublime-commands, .sublime-menu, ...) would be read from the Lib folder.

[wbond]

There is nothing precluding a dependency creating a base command class and having plugins extend that. Sublime Text just won’t directly instantiate the command from the dependency, which makes sense.

[rchl]

It's not about that in that specific case though. The dependency wants to expose its settings to the user. It's not the job of a plugin to do that because there can be multiple plugins using a dependency and should all of them do that?

[wbond]

I’m sorry?

Dependencies won’t be able to do these things anymore, you’ll have to do something else.

The simplest options are:

1. Vendor things
2. Use conventions. I.e. have 5 plugins all decide to respect User/mdpopups.sublime-settings and use some function mdpopups.get_default() for the base configuration.

The sooner this process starts, the less likely things will get broken.

[wbond]

I feel like we should rename dependencies to “libraries” during this “rewrite”. It will open up the term dependencies for future use if someone wants to make packages depend on each other, and it will make their intent more clear.

[FichteFoll]

Hard +1 on the rename. That will make their role way more obvious and people won't expect to be able to depend on other packages either.

Throughout the last 4 years or so (basically ever since ST3 got the Lib folder) I have strongly discouraged anyone adding a dependency to the channel to depend on either loader.py or ST loading bundled resource files, meaning we should have very few dependencies doing so. And indeed, it appears only mdpopups is (didn't check myself).

If such depenedency wants to streamline its configuration, it should find a different way to do that, such as by providing a command that performs the necessary UI setup for user configuration and then suggesting dependant packages to include a certain snippet in some menu file (that will deduplicate if every level is bound by an id). Unfortunately, that won't do for the command palette, where packages would need to either namespace these commands or the user would end up with multiple entries.

Maybe sublime_plugin is interested in exposing an actual endpoint for registering custom commands like that?

[deathaxe]

I was keen enough to push the open PR towards this direction, already, as it just feels right this way.

[rwols]

What's the status on this now?

[deathaxe]

According to https://github.com/wbond/package_control/tree/wbond-four-point-oh in the middle of heavy changes to rework infrastructure, I'd say.

[deathaxe]

With 1b34ff1ea12af2ac4a8109f6963e880a11d96b82 four-point-oh has reached a state of being able to install/upgrade/remove ST3-style dependencies into Lib/ folder.