Opt-in to python 3.8 plugin host

[deathaxe]

This commit adds .python-version file to migrate the plugin to python 3.8

[facelessuser]

How does this work with dependencies that I have not migrated to 3.8?

[facelessuser]

While I know we can opt a package into 3.8, is there official documentation somewhere that explains the rules of how 3.8 packages interact with 3.3 dependencies and guides to properly upgrade dependencies to 3.8? My original plan was to update dependencies to 3.8 first and then upgrade packages. Everytime I've considered doing that, I realize I can't find documentation as to how we distinguish a dependency for 3.8 (I think I asked in discord once, and I can no longer find that conversation). There seems to be no guide on the Package Control site.

I'm okay upgrading some packages first, assuming they can work with 3.3 dependencies (I assume only pure Python dependencies have a chance of working), but I do want to get the dependencies updated as well.

[deathaxe]

Unfortunatelly we don't have access to packagecontrol.io to add or update any documentation.

A package running on a certain plugin host (e.g. py3.8) uses dependencies (aka. libraries) which are installed in the same environment. Actually I have been running ColorHelper on python 3.8 for quite a while now.

Dependencies (or now just called libraries) are therefore shipped by a 2nd channel. Any python 3.8 library would therefore need to be registered at https://github.com/packagecontrol/channel/.

By default all python 3.3 dependencies, which seemed compatible with py38 have just been opted-in to it, so by default most legacy dependencies are available in python 3.8 runtime by default.

Actually many of them have already been migrated to directly use latest releases from pypi.org

Package Control - even though not yet resolving requirements.txt - happily installs any *.whl file provided. It just needs to be registered in said channel and calling package needs to specify all required packages.

For those who love to keep working with the legacy dependency format, https://github.com/packagecontrol/example-dependency new options have been added to provide binaries for py38 and ST4 only.

Legacy dependencies are converted to normal python packages and are installed to "$st-data-dir/Lib/python33" or "$st-data-dir/Lib/python38" as if pip would have installed them there.

[facelessuser]

So, your saying mdpopups has been opted into 3.8 Even though I have not verified everything is compatible? I guess it seems to be working, but that could have caused me a lot of work.

So currently it works on both 3.3 and 3.8?

It seems using st4 can allow me to differentiate and then kill off support for 3.3?

[facelessuser]

How is Mac M1, Mx, etc. support handled? Is it specified as arm?

[facelessuser]

This should allow us to more easily upgrade the internal color library as I won't have to backport things, so I think this is a good thing. It is likely I can start doing this with most of the plugins I support except those that use regex as those our compiled libraries. I'll need to unpack the latest 3.8 wheels and make sure there is nothing special I need to do for mac MX chips.

[deathaxe]

... though I have not verified everything is compatible?

I have used mdpopups together with GitGutter, LSP and other packages on python 3.8 for a very long time since working on PC4 without noticing any issues at all. That's why judged it safe to enable it for python 3.8

So currently it works on both 3.3 and 3.8?

Yes it does and it should keep doing so.

It seems using st4 can allow me to differentiate and then kill off support for 3.3?

At least current version should continue being available for python 3.3. But yes, variants for python 3.8 can move on.

How is Mac M1, Mx, etc. support handled?

ST is shipped as universal app and so should packages/libraries with compiled libraries. I've already seen various python packages doing so. They are named like ...-universal2....

In general ST reports x64 on Intel Macs and when running via rosetta on Mx, while arm64 is reported when running natively on a Mx.

[facelessuser]

@gir-bot lgtm

[facelessuser]

@deathaxe I guess I'll be a little more direct with my question. Does using the st4- prefix force a dependency to use Python 3.8? What I'm really getting at is whether I have to keep backporting internal libs in mdpopups to Python 3.3 moving forward. This is the current drain on me at the present, and the thing I want to move past the most. It hinders me from updating anything forward because the backporting can be such a pain.

My hope is that anyone who has not opted into Py 3.8 will be frozen in time with mdpopups on 3.3, but everyone opting into Py 3.8 can get some much needed updates.

[facelessuser]

@deathaxe

This is what I'm getting with the updated change:

Package Control: The library "sublime_lib" is not available for Python 3.8
Package Control: The library "bracex" is not available for Python 3.8
Package Control: The library "wcmatch" is not available for Python 3.8
Package Control: The library "lsp_utils" is not available for Python 3.8
Package Control: The library "typing_extensions" is not available for Python 3.8
Package Control: The library "mdpopups" is not available for Python 3.8
Package Control: The library "PyYAML" is not available for Python 3.8

Am I missing something? I'm afraid I may have to revert if I cannot get this to work locally.

[deathaxe]

Appears, channel_v4 is missing. In case you've customized your "channels" setting, check if the first two entries match the default.

{
    "channels": [
        // channel_v4 for python 3.8 compatible libraries to enable plugins
        // to migrate to python 3.8 until packagecontrol.io supports new scheme.
        // Note: Must always be located before default channel in the list!
        "https://packagecontrol.github.io/channel/channel_v4.json",
        // default channel for packages
        "https://packagecontrol.io/channel_v3.json"
    ],
}

[facelessuser]

Yep, I just figured that out myself

[deathaxe]

Does using the st4- prefix force a dependency to use Python 3.8?

If you want to keep going with legacy dependency format, there are 2 options:

1. Create a st4_py38/<name> next to the existing st3/<name>. The former will be used on python 3.8 plugin host and the latter will be installed on ST3 and ST4 python 3.3. If you want to leave ST3 behind but still add updates to ST4 on python 3.3, you could also add an st4 or st4_py33 folder.

2. To reduce download size, you could of cause also create a new branch just for python 3.8 releases. I'd however suggest tag prefix like py38- as the current release will also be installed on ST4's python 3.3 runtime when requested by a package. So st4- prefix could be somewhat confusing, maybe.

   In that case the repository would need to be adjusted to:

   {
   "name": "mdpopups",
   "description": "Markdown Popups for Sublime",
   "author": "facelessuser",
   "issues": "https://github.com/facelessuser/sublime-markdown-popups/issues",
   "releases": [
       {
           "base": "https://github.com/facelessuser/sublime-markdown-popups",
           "python_versions": ["3.3"],
           "tags": "py33-"
       },
       {
           "base": "https://github.com/facelessuser/sublime-markdown-popups",
           "python_versions": ["3.8"],
           "tags": "py38-"
       }
   ]
   }

If adding a build step would be acceptable, you could also create a *.whl file for each library and deploy as download asset via github releases or any other hosting mechanism. In that case, we could adjust the repository entry to an asset based release.

Assuming whl files being deployed under https://github.com/facelessuser/sublime-markdown-popups/releases/ the entries would look like:

{
    "name": "mdpopups",
    "description": "Markdown Popups for Sublime",
    "author": "facelessuser",
    "issues": "https://github.com/facelessuser/sublime-markdown-popups/issues",
    "releases": [
        {
            "asset": "mdpopups-*-py33-none-any.whl",
            "python_versions": ["3.3"],
            "tags": "py33-"
        },
        {
            "asset": "mdpopups-*-py38-none-any.whl",
            "python_versions": ["3.8"],
            "tags": "py38-"
        }
    ]
},

[deathaxe]

Actually Will's original idea was to completely move libraries to wheels. Hence it would be appriciated if someone could teach community how to efficiently create and deploy wheels for ST.

[facelessuser]

To reduce download size, you could of cause also create a new branch just for python 3.8 releases. I'd however suggest tag prefix like py38- as the current release will also be installed on ST4's python 3.3 runtime when requested by a package. So st4- prefix could be somewhat confusing, maybe.

Hmm, this is intriguing. So I can migrate the current master to Python 3.8 and just tag all new releases with the prefix py38-. This would probably be the way I'd like to go. I guess then I can just migrate all internal folders to a simple st4- folder orst4-<arch> folder for those that require architecture specifiers.

I think this may be the way to go. I'll have to make py33- tags on all my current dependencies, and then make the submission to Package Control to start using this convention. Once in place, I can make the transition. Oh...I'll have to make the py38- tag prior as well so I don't break packages relying on a py38- environment, but once both have such tags, I should be able to transition.

[deathaxe]

Note that arch specific sub directories always need the python specifier as well st4_arm64 would be invalid. It would need to be st4_py38_arm64. Arch specific releases are considered containing compiled libraries, which always depend on python version.