Natron 2.5 release w/ Python 3 (track progress)

[devernay]

This issue tracks Python3 support progress Feel free to edit this issue, and use comments for discussion.

Target release

Release 2.5 will use Python3.

Builds are already available for Linux and macOS: https://github.com/NatronGitHub/Natron/releases/tag/v2.5.0-alpha.2

Windows support is the main thing to fix before releasing 2.5.0

Bugs introduced by Python3

None identified

OS Support

Windows

• [X] Build (MinGW)
• Build (MSVC?) maybe for 2.6
• [x] Binary distribution
• [ ] pip install future in 2.4.4 https://github.com/NatronGitHub/Natron/issues/828#issuecomment-1332719283

Linux

• [X] Build (CentOS7+DTS11, docker image)
• [X] Binary distribution (installer)
• [X] Binary distribution (no installer)
• [x] pip install future in 2.4.4

macOS

• [X] Build (10.9)
• [X] Build (10.15)
• [x] Build (12.x Intel)
• [x] Build (12.x M1) (not supported by Qt4)
• [X] Binary distribution (Intel)
• [x] Binary distribution (M1) (not supported by Qt4)
• [x] pip install future in 2.4.4 10.6
• [x] pip install future in 2.4.4 10.9
• [x] pip install future in 2.4.4 10.15
• [X] pip install future in 2.4.4 12

[rodlie]

I started the transition to latest MSYS2 today. Should be finished in a couple of days.

[rodlie]

I'm now able to build (need #831 and #830) Natron and plug-ins on latest MSYS2 (from today) with GCC 12.1 and Python 3.10.

Will continue with the packaging a bit later today.

[devernay]

Any progress on that @rodlie ?

[rodlie]

I can dedicate Saturday to finish the Windows stuff.

[devernay]

you don't /have/ to!

[rodlie]

So, everything builds and seems to run, however:

• I need to make a python embed zip, as we can't download and use the embed zip from python.org
• pip on Windows will most likely not work properly, if a package needs building it will fail (some prebuilt may also fail)
  • Not an regression, as it's always been like this. python-mingw don't mix with python-msvc.

[devernay]

I need to make a python embed zip, as we can't download and use the embed zip from python.org

why doesn't this zip work? making your own is rather risky. We only take the python39.zip file from it, which contains portable byte-compiled files. We use the same on macOS and Linux, why not on Win32?

[devernay]

I tagged version 2.5.0, and started producing binaries

[rodlie]

why doesn't this zip work? making your own is rather risky. We only take the python39.zip file from it, which contains portable byte-compiled files. We use the same on macOS and Linux, why not on Win32?

It's built for MSVC and includes and depends on libraries (sqlite/openssl) that conflicts with the libraries we ship in Natron. I don't see any risk as I produce the same zip as upstream, but linked properly (and built with MinGW).

I tagged version 2.5.0, and started producing binaries

Ok, I will try to get a Windows build packaged tomorrow, I don't think I have enough time to finish it today.

[devernay]

There's no binaries in the zip-inside-the-zip (check out how we extract it in zip-python.zip), so no risk of binary conflict! As I said, we are already using the same zip on Linux and macOS

[devernay]

Here's the content of python310.zip (the LINUX one, which I use on macOS!). As you see: no binary, only portable bytecode produced by python itself. So no risk of compiler or library incompatibilities python310-list.txt

[rodlie]

Yes, sorry I misunderstood. Something with that upstream zip does not work, as trying to use that file breaks the Python we use in Natron. It's been a while since I last did anything with this, so let me double check and report back what the problem was.

[rodlie]

Works with Python 3.10 :+1: , but not with 3.9 or lower (yes I used the correct zip). Might be an MSYS2 bug with older Python's (remember MSYS2 uses a fork of Python).

Then I might get something ready today.

[devernay]

When you have time, can you please also rebuild 2.4.4 with the script that fixes qtpy: https://github.com/NatronGitHub/Natron/commit/4ee9541824e26af51a33be4109fc548f005adc7c

With that fix, "from qtpy import QtCore" should work in "NatronRenderer -t".

Not a requirement, since qtpy is supposed to be a Natron 2.5 feature, but that will help the users in their transition to Qt5

[rodlie]

Ok, will do, currently having issues cloning from github so can't run the build scripts at the moment.

Btw, get a lot of warnings in RB-2.5/plugins: include/c++/12.1.0/x86_64-w64-mingw32/bits/c++config.h:642:2: warning: #warning "__STRICT_ANSI__ seems to have been undefined; this is not supported" [-Wcpp] 642 | #warning "__STRICT_ANSI__ seems to have been undefined; this is not supported"

this seems to be the reason for libs/ceres not building: error: 'j0' was not declared in this scope

https://stackoverflow.com/questions/42231558/mingw-c-wont-compile-j0-funciton

[devernay]

wait a bit till I give the green light to rebuild 2.4.4, I'm looking into these issues.

as for those CPPFLAGS:

• it's been added a long time ago, and the reason is in the comments. If if works without those flags feel free to remove them from openfx (both in Support and Examples) https://github.com/NatronGitHub/openfx/commit/002cf0b8a0c6022e3abc13e00fa5f5197b8e1099
• they're not used for building ceres, so the cause of ceres not building must be something else

[devernay]

ok build issues on linux and macOS are fixed, I re-created the 2.4.4 and 2.5.0 binaries. You can go with windows. The most important fix was for qtpy, which didn't work on Qt4

[devernay]

all linux and macOS 2.5.0 binaries were uploaded and ready for release

[rodlie]

I'm waiting on a new snapshot build, fixed the build issues, will see how far the python stuff goes. After that I need to update a couple of packages (oiio etc) then run a release build. Unsure when done, probably tomorrow (as it's 21:15 here).

[rodlie]

Something is still off with the python310.zip I got from upstream.

pip will not work get-pip.py:

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip

If I replace the python310.zip with the one I made it works.

Collecting pip
  Using cached pip-22.3.1-py3-none-any.whl (2.1 MB)
Collecting setuptools
  Downloading setuptools-65.5.1-py3-none-any.whl (1.2 MB)
     ---------------------------------------- 1.2/1.2 MB 15.7 MB/s eta 0:00:00
Collecting wheel
  Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
Installing collected packages: wheel, setuptools, pip
  WARNING: The script wheel.exe is installed in 'C:/msys64-20220603/home/rodlie/Natron-workspace/tmp64/tmp_deploy/Natron-RB-2.5-202211171941-e0a8067-Windows-x86_64/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts pip.exe, pip3.10.exe and pip3.exe are installed in 'C:/msys64-20220603/home/rodlie/Natron-workspace/tmp64/tmp_deploy/Natron-RB-2.5-202211171941-e0a8067-Windows-x86_64/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-22.3.1 setuptools-65.5.1 wheel-0.38.4

[rodlie]

So, finally able to produce a RB-2.5 snapshot build. Still need to update some libs (oiio etc), and go through my build changes and commit.

I do however get two new unit test failures.

GNUVolador

I assume this is because I now have a modern version of librsvg(?).

PIK

Don't know why this failed, will need to go through the logs.

[devernay]

Those failures also happen on macOS, but not on Linux. Both failures are new to 2.5.0 but were absent from 2.5.0alpha2. I'll try to give it a look, but the main difference is the Python version (3.9 on Linux, 3.10 on 2.5.0 macOS, 3.9 on 2.5.0a2 macOS)

[devernay]

On macOS, 2.5.0alpha2 had librsvg 2.52.4, 2.5.0 has 2.54.5

[rodlie]

I currently have issues with deadlocks in the unit tests. I have tried upgrading/downgrading OIIO etc, but nothing seems to work. This is an regression compared to v2.4.

[rodlie]

I uploaded the 2.5.0 build. Seems ok (disregarding failed tests), however the deadlocks in the unit tests are concerning and Natron may not close properly in some circumstances.

I will try to improve the new sdk sometime this week(end).

[devernay]

the only major difference is python 3. 2.5 probably still compiles with python 2, can you give it a test?

[rodlie]

I suspect this being a MSYS/MinGW issue, but will try to do a build with py2 (if possible).

[bonalex01dev]

Hi, I tested the 2,5 pre windows build with success: It is at least as stable as 2.4.3 Maybe a little better on the shadertoy side. (I happened to just discover that just disable/enable a shadertoy can bring it back to life) As before we get better stability with "process whole frame" (which Nuke seems to be making the norm in recent versions). Thank you for the hard work!

Some questions: Is it time to port community scripts to python 3? Is it time to advertise this new version? (pixls /discord)

[devernay]

OK I fixed the GNUVolador issue. That was a major issue with Python >= 3.10 (couldn't use a double-values expression for an int parameter): https://github.com/NatronGitHub/Natron/commit/54415e76e14a55a924cd64b5b26c27963a851085

Now working on the PIK failure

[devernay]

@bonalex01dev

Is it time to port community scripts to python 3?

Actually, it's even better to make them 2-3 compatible, using these guidelines: https://python-future.org/compatible_idioms.html

Is it time to advertise this new version? (pixls /discord)

Not until we mark 2.5.0 as a release (it's currently pre-release)

[devernay]

Both tests are passing now, you can start building for windows, @rodlie ! I'm updating the Linux SDK with python 3.11 before I do the linux builds

[rodlie]

Sorry for the delay (been sick), GNUVolador still fails here (release build).

[devernay]

you mean fails as before (which is just a large image difference due to a different version of librsvg I think), or fails as in the image above? I fixed the root cause for the wings not animating, so I don't see how it could fail only on windows.

[rodlie]

Sorry for not being specific. The test fails, but it's not the same: Probably the librsvg version as you say.

Will upload the new build (still deadlocks during unit tests, will try to update to latest msys on a different instance, don't want to break anything yet).

[devernay]

So are we all good? I think all lights are green for the release.

Then we can focus on the migration to Qt5 (2.6) - and maybe switch the windows build to using MSVC + vcpkg

[rodlie]

So are we all good?

Yeah, the Windows build seems to work. Limited testing on my part (tested on a couple of workstations with some real life projects, don't see any regression compared to v2.4).

Then we can focus on the migration to Qt5 (2.6)

Ok. Will need some tweaking on Windows at least, but not anything major.

and maybe switch the windows build to using MSVC + vcpkg

That's a big maybe ;) (at least this year).

[devernay]

released!

[bonalex01dev]

Hi, to start updating the community scripts, I was able to load the "future" module in natron 2.5 with: natron-python.exe -m pip install future

but in natron 2.4.3 the same command throws an error: distutils.errors.DistutilsPlatformError: VC 6.0 is not supported by this module

My understanding is we need it on both versions to make scripts py2 py3 compatibles.

Do you have a working version of "future" for python 2.7 windows ? Or another solution ?

Thanks again for the release!

[devernay]

isn't future bundled with python? BTW I fixed issues with natron-plugins, they now load in 2.5

[devernay]

No future... I forgot to install it. We will probably release 2.4.5 and 2.5.1 with future installed

[devernay]

Anyway, I don't think there were any Python2-specific constructs in the community scripts. I ran "2to3 ." in the natron-plugins directory, and the only scrit that was affected is Draw/Audio_VLC/Audio_VLCExt.py which contains 2 prints (from this commit by @Windows81 https://github.com/NatronGitHub/natron-plugins/commit/ddde06429fd180f8e5540b8a51df59922d050b87 )

[Windows81]

Nice to see Natron finally updating to Python 3! I'm glad none of the community plugins had real compatibility problems. I do hear that some built-in libraries available in Python 2 aren't available in 3.11 (no problems there as well). We're quite a conservative bunch!

[devernay]

The only issue was illegal UTF-8 characters in Python files. Fixed in https://github.com/NatronGitHub/natron-plugins/pull/18

[devernay]

@rodlie could you publish new 2.4.4 and 2.5.0 Windows binaries with "future" installed? I fixed the scripts here https://github.com/NatronGitHub/Natron/commit/a303f0f80532d0f775a63869cbcb9228e9c974f0

[rodlie]

Ok, will need to wait until I'm home (at work).

[rodlie]

Just uploaded v2.5.0 with future. Will start a new v2.4.4 build in a couple of hours (different computer).

[rodlie]

Can't install future for v2.4 using pip:

Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     |################################| 829 kB 6.6 MB/s
    ERROR: Command errored out with exit status 1:
     command: C:/msys64-20180531/home/Natron/workspace/tmp64/tmp_deploy/Natron-2.4.4-Windows-x86_64/bin/natron-python.exe -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'c:/msys64-20180531/tmp/pip-install-odxrp_/future/setup.py'"'"'; __file__='"'"'c:/msys64-20180531/tmp/pip-install-odxrp_/future/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base c:/msys64-20180531/tmp/pip-pip-egg-info-mdydis
         cwd: c:/msys64-20180531/tmp/pip-install-odxrp_/future/
    Complete output (13 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:/msys64-20180531/home/Natron/workspace/tmp64/tmp_deploy/Natron-2.4.4-Windows-x86_64/lib/python2.7/site-packages/setuptools/__init__.py", line 245, in <module>
        monkey.patch_all()
      File "C:/msys64-20180531/home/Natron/workspace/tmp64/tmp_deploy/Natron-2.4.4-Windows-x86_64/lib/python2.7/site-packages/setuptools/monkey.py", line 101, in patch_all
        patch_for_msvc_specialized_compiler()
      File "C:/msys64-20180531/home/Natron/workspace/tmp64/tmp_deploy/Natron-2.4.4-Windows-x86_64/lib/python2.7/site-packages/setuptools/monkey.py", line 164, in patch_for_msvc_specialized_compiler
        patch_func(*msvc9('find_vcvarsall'))
      File "C:/msys64-20180531/home/Natron/workspace/tmp64/tmp_deploy/Natron-2.4.4-Windows-x86_64/lib/python2.7/site-packages/setuptools/monkey.py", line 151, in patch_params
        mod = import_module(mod_name)
      File "./importlib/__init__.py", line 37, in import_module
      File "./distutils/msvc9compiler.py", line 306, in <module>
    distutils.errors.DistutilsPlatformError: VC 6.0 is not supported by this module
    ----------------------------------------

[devernay]

@rodlie did you try

pip uninstall setuptools
pacman -S mingw-w64-x86_x64-python-setuptools

https://stackoverflow.com/a/55554854

[devernay]

reopening to fix the "pip install future"

[rodlie]

I have mingw setuptools installed, but setuptools was installed by pip since the mingw setuptools was not in the tmp deploy folder. Should have a new build done within an hour.