linux/macos support in vsrepo.py

[Stefan-Olt]

I added support for Linux / macOS support in vsrepo. The changes are rather small:

• The function detect_target will return a target string (like win32/win64 on Windows). Besides Windows it can currently detect Linux and macOS on x86_64 and arm64/aarch64. It is used where win32/win64 was previously decided with if/else. As macOS identifies itself as Darwin, this term is used in the string instead of macOS. On Linux it currently just assumes that glibc is used (this is included in the string because the binaries are not compatible to other libc implementations (that are rarely used on PC)).
• Paths detection needed to be modified: For portable it's the same for all targets, for other paths I did this:
  • The dir for vspackages3.json is stored $(HOME)/.config/vsrepo
  • The binary plugins are stored in /usr/local/lib/vapoursynth if run as root, otherwise in $(HOME)/.local/lib/vapoursynth
  • The Python path is the same as in Windows if run as user, if run as root getsitepackages() is used to get a system-wide path. If included in the list, the first entry that starts with /usr/local is used, because that dir should not be managed by the system packet manager, otherwise the first entry will be used
• For Linux/macOS the 7z command is just 7z

Currently, if you want to install a binary plug-in on macOS/Linux the completely correct message that there is no binary for that target is shown (was already implemented for win32/win64). I already created a experimental GitHub repo, that will build (and test) Linux binaries of plug-ins using an older glibc with their dependencies statically linked, for high compatibility with many distributions. Also macOS x86_64 ("Intel") binaries are created, no ARM ("Apple Silicon") yet, because many plug-ins need at least minor adjustments to correctly build on ARM. The method to build and test the plugins are stored in json files. My suggestion would be, that during the build process, also a json is created, that contains the information for vsrupdate to include them in the json files for the plugins. This way, vsrupdate would just check all releases for the existence of that json file and incorporate that.

[theChaosCoder]

I just tried it in my old ubuntu 18 VM. I used this release data for ffms2:

{
          "version": "5.0",
          "published": "2024-05-28T19:53:13Z",
          "linux-glibc-x86_64": {
            "url": "https://github.com/Stefan-Olt/vs-plugin-build/releases/download/ffms2-227a29cbcd8ac0fc2926a266d099a3c9feeda493/FFmpegSource2-5.0-linux-glibc-x86_64.zip",
            "files": {
              "libffms2.so": [
                "libffms2.so",
                "7c51de8036d1c62b9bca23b1ed9cc90ec118f361d4ff3c7c5ac0ef6acf554a02"
              ]
            }
          }
        },

If think the script also need a header like #!/usr/bin/env python3 bcs otherwise ./vsrepo update does not call python on my system but something else...

python3 vsrepo install ffms2 ran fine. The file was "installed"/copied into my home folder under .local/lib/vapoursynth. But after that the script failed with a syntax error in update_genstubs()

vsgenstubs4/init.py", line 273
    def __gt__(self, x: 'PluginMeta', /) -> bool: return self._str_('__gt__', x)  # type: ignore[override]

My suggestion would be, that during the build process, also a json is created, that contains the information for vsrupdate to include them in the json files for the plugins. This way, vsrupdate would just check all releases for the existence of that json file and incorporate that. You mean generate the whole releases block for a plugin during the build process? like the ffms2 example above?

I think we could make it work with vsrupdate with minimal code changes. Look at https://api.github.com/repos/Stefan-Olt/vs-plugin-build/releases Every release item has a tag_name. "tag_name": "ffms2-257712fe8f3f103bfc94d236e8efb02dd18f1bcd", "tag_name": "nnedi3-11f830025f46ea2a045e148395db48d771edba19", etc. If we generate the tag always like this tag_name = plugin_namespace-randomID we could easily filter out the unwanted releases. No custom json files during the build process necessary.

Or perhaps we also need the os info? Like this: tag_name = plugin_namespace-os-randomID => "tag_name": "ffms2-linux-257712fbcd"

[Stefan-Olt]

If think the script also need a header like #!/usr/bin/env python3 bcs otherwise ./vsrepo update does not call python on my system but something else...

I didn't add that on purpose, because you would still need to call vsrepo.py. My idea would have been to have shell script named vsrepo that runs python3 vsrepo.py [ARGS]. This could also handle the issue with multiple Python versions: If for example on macOS you install Python normally and install VapourSynth using brew, you'll end up with two Python versions, but VapourSynth is only installed in the one provided by brew. But scripts will run the other one by default. The shell script could be created during installation and include the full path to the correct Python binary.

python3 vsrepo install ffms2 ran fine. The file was "installed"/copied into my home folder under .local/lib/vapoursynth. But after that the script failed with a syntax error in update_genstubs()

I tried exactly the same (copied the section into the vspackages3.json), no issue at all (on Ubuntu 22.04). Ubuntu 18 came with Python 3.6, maybe that's too old for vsrepo. Do you know if current vsrepo works with Python 3.6 on Windows?

I think we could make it work with vsrupdate with minimal code changes. Look at https://api.github.com/repos/Stefan-Olt/vs-plugin-build/releases Every release item has a tag_name. "tag_name": "ffms2-257712fe8f3f103bfc94d236e8efb02dd18f1bcd", "tag_name": "nnedi3-11f830025f46ea2a045e148395db48d771edba19", etc. If we generate the tag always like this tag_name = plugin_namespace-randomID we could easily filter out the unwanted releases. No custom json files during the build process necessary.

Or perhaps we also need the os info? Like this: tag_name = plugin_namespace-os-randomID => "tag_name": "ffms2-linux-257712fbcd"

That's no problem at all, I have the namespace stored in json with the build definition, I can add that to the tag. The OS type is included in the filename (filename is always name-version-os.zip), so it's not needed. But maybe the tag should contain all information to make it easier to parse, in case the plugin name or the version contains dashes. tags are allowed to be hierarchical, so plugin_namespace/version/os/randomid should work.

[theChaosCoder]

For R69 python 3.12 and 3.8 are supported on windows. I think cython on windows has some limitations.

[Stefan-Olt]

For R69 python 3.12 and 3.8 are supported on windows. I think cython on windows has some limitations.

The question is, is the issue you experienced caused by a old Python version, or something else? If you have a old VapourSynth installation with Python 3.6 on Windows (like R44) and copy latest vsrepo into it, would it work? A syntax error sounds like a Python version problem to me, but I'm not sure.

[theChaosCoder]

More tests: python3 vsrepo.py gendistinfo runs without errors. It calls update_genstubs(). The problem is rebuild_distinfo(), the import fails for me from vsgenstubs4 import main as genstubs4

Python 3.6.9:

>>> from vsgenstubs4 import main as genstubs4
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vmtest/Downloads/vsrepo-master/vsgenstubs4/__init__.py", line 1, in <module>
    from .init import main  # noqa: F401
  File "/home/vmtest/Downloads/vsrepo-master/vsgenstubs4/init.py", line 273
    def __gt__(self, x: 'PluginMeta', /) -> bool: return self._str_('__gt__', x)  # type: ignore[override]
                                      ^
SyntaxError: invalid syntax

I guess it uses some newer python override syntax in the PluginMeta class? I don't code much in python.

Never tested new vsrepo with really old VS versions on windows. I think it should work or maybe it will have the same problem :D

[Stefan-Olt]

Python 3.6.9:

>>> from vsgenstubs4 import main as genstubs4
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vmtest/Downloads/vsrepo-master/vsgenstubs4/__init__.py", line 1, in <module>
    from .init import main  # noqa: F401
  File "/home/vmtest/Downloads/vsrepo-master/vsgenstubs4/init.py", line 273
    def __gt__(self, x: 'PluginMeta', /) -> bool: return self._str_('__gt__', x)  # type: ignore[override]
                                      ^
SyntaxError: invalid syntax

I guess it uses some newer python override syntax in the PluginMeta class? I don't code much in python.

It's related to the Python version, the slash indicates that arguments are positional only, this feature was introduced in Python 3.8: https://peps.python.org/pep-0570/ So this has nothing to do with my changes (I did not touch that file at all) or Linux/Windows in general, it's just that vsrepo requires at least Python 3.8

[Stefan-Olt]

I updated the github runners so they will create tags like this: vsplugin/{identifier}/{version}/{target}/{utc_datetime} Is this a good tag system for vsrupdate?

[Stefan-Olt]

Any progress on testing this? Anything that I can do?

[theChaosCoder]

Looks good from my side.

I also wanted to add update/tag support in vsrupdate, but I'm currently too busy with work stuff...

There is "updatemode": "git-commits", so I was thinking adding updatemode = git-tags or something like that + maybe an array with git tags wich will act as a filter based on your generated tagging format {identifier}/{version}/{target}/{utc_datetime}.

[myrsloik]

Looks reasonable and a lot less changes than I thought. Go ahead and merge it @theChaosCoder if you have the time to actually add some plugins.

[theChaosCoder]

Should we bump file-format to 4 then?

[myrsloik]

Should we bump file-format to 4 then?

I guess so.

[Stefan-Olt]

Great to hear! I did not change any file-format, should be fully backward compatible. But maybe to indicate that target can now be something else than win32 or win64

I will add to my build repo an automatic update system similar to VSRUpdate.py, to make updating the plugins somewhat automatic and also to generate the build json mostly automatically for adding new plugins

[theChaosCoder]

I added releases for ffms2, mvtools, nnedi & colorbars so people can test it.