Natron 2.5 release w/ Python 3 (track progress)

devernay commented 2 years ago

This issue tracks Python3 support progress Feel free to edit this issue, and use comments for discussion.

Target release

Release 2.5 will use Python3.

Builds are already available for Linux and macOS: https://github.com/NatronGitHub/Natron/releases/tag/v2.5.0-alpha.2

Windows support is the main thing to fix before releasing 2.5.0

Bugs introduced by Python3

None identified

OS Support

Windows

[X] Build (MinGW)
~~Build (MSVC?)~~ maybe for 2.6
[x] Binary distribution
[ ] pip install future in 2.4.4 https://github.com/NatronGitHub/Natron/issues/828#issuecomment-1332719283

Linux

[X] Build (CentOS7+DTS11, docker image)
[X] Binary distribution (installer)
[X] Binary distribution (no installer)
[x] pip install future in 2.4.4

macOS

[X] Build (10.9)
[X] Build (10.15)
[x] Build (12.x Intel)
[x] ~~Build (12.x M1)~~ (not supported by Qt4)
[X] Binary distribution (Intel)
[x] ~~Binary distribution (M1)~~ (not supported by Qt4)
[x] pip install future in 2.4.4 10.6
[x] pip install future in 2.4.4 10.9
[x] pip install future in 2.4.4 10.15
[X] pip install future in 2.4.4 12

rodlie commented 2 years ago

I started the transition to latest MSYS2 today. Should be finished in a couple of days.

rodlie commented 2 years ago

I'm now able to build (need #831 and #830) Natron and plug-ins on latest MSYS2 (from today) with GCC 12.1 and Python 3.10.

Will continue with the packaging a bit later today.

devernay commented 2 years ago

Any progress on that @rodlie ?

rodlie commented 2 years ago

I can dedicate Saturday to finish the Windows stuff.

devernay commented 2 years ago

you don't /have/ to!

rodlie commented 2 years ago

So, everything builds and seems to run, however:

I need to make a python embed zip, as we can't download and use the embed zip from python.org
pip on Windows will most likely not work properly, if a package needs building it will fail (some prebuilt may also fail)
- Not an regression, as it's always been like this. python-mingw don't mix with python-msvc.

devernay commented 2 years ago

I need to make a python embed zip, as we can't download and use the embed zip from python.org

why doesn't this zip work? making your own is rather risky. We only take the python39.zip file from it, which contains portable byte-compiled files. We use the same on macOS and Linux, why not on Win32?

devernay commented 2 years ago

I tagged version 2.5.0, and started producing binaries

rodlie commented 2 years ago

why doesn't this zip work? making your own is rather risky. We only take the python39.zip file from it, which contains portable byte-compiled files. We use the same on macOS and Linux, why not on Win32?

It's built for MSVC and includes and depends on libraries (sqlite/openssl) that conflicts with the libraries we ship in Natron. I don't see any risk as I produce the same zip as upstream, but linked properly (and built with MinGW).

I tagged version 2.5.0, and started producing binaries

Ok, I will try to get a Windows build packaged tomorrow, I don't think I have enough time to finish it today.

devernay commented 2 years ago

There's no binaries in the zip-inside-the-zip (check out how we extract it in zip-python.zip), so no risk of binary conflict! As I said, we are already using the same zip on Linux and macOS

devernay commented 2 years ago

Here's the content of python310.zip (the LINUX one, which I use on macOS!). As you see: no binary, only portable bytecode produced by python itself. So no risk of compiler or library incompatibilities python310-list.txt

rodlie commented 2 years ago

Yes, sorry I misunderstood. Something with that upstream zip does not work, as trying to use that file breaks the Python we use in Natron. It's been a while since I last did anything with this, so let me double check and report back what the problem was.

rodlie commented 2 years ago

Works with Python 3.10 :+1: , but not with 3.9 or lower (yes I used the correct zip). Might be an MSYS2 bug with older Python's (remember MSYS2 uses a fork of Python).

Then I might get something ready today.

devernay commented 2 years ago

When you have time, can you please also rebuild 2.4.4 with the script that fixes qtpy: https://github.com/NatronGitHub/Natron/commit/4ee9541824e26af51a33be4109fc548f005adc7c

With that fix, "from qtpy import QtCore" should work in "NatronRenderer -t".

Not a requirement, since qtpy is supposed to be a Natron 2.5 feature, but that will help the users in their transition to Qt5

rodlie commented 2 years ago

Ok, will do, currently having issues cloning from github so can't run the build scripts at the moment.

Btw, get a lot of warnings in RB-2.5/plugins: include/c++/12.1.0/x86_64-w64-mingw32/bits/c++config.h:642:2: warning: #warning "__STRICT_ANSI__ seems to have been undefined; this is not supported" [-Wcpp] 642 | #warning "__STRICT_ANSI__ seems to have been undefined; this is not supported"

this seems to be the reason for libs/ceres not building: error: 'j0' was not declared in this scope

https://stackoverflow.com/questions/42231558/mingw-c-wont-compile-j0-funciton

devernay commented 2 years ago

wait a bit till I give the green light to rebuild 2.4.4, I'm looking into these issues.

as for those CPPFLAGS:

it's been added a long time ago, and the reason is in the comments. If if works without those flags feel free to remove them from openfx (both in Support and Examples) https://github.com/NatronGitHub/openfx/commit/002cf0b8a0c6022e3abc13e00fa5f5197b8e1099
they're not used for building ceres, so the cause of ceres not building must be something else

devernay commented 2 years ago

ok build issues on linux and macOS are fixed, I re-created the 2.4.4 and 2.5.0 binaries. You can go with windows. The most important fix was for qtpy, which didn't work on Qt4

devernay commented 2 years ago

all linux and macOS 2.5.0 binaries were uploaded and ready for release

rodlie commented 2 years ago

I'm waiting on a new snapshot build, fixed the build issues, will see how far the python stuff goes. After that I need to update a couple of packages (oiio etc) then run a release build. Unsure when done, probably tomorrow (as it's 21:15 here).

rodlie commented 2 years ago

Something is still off with the python310.zip I got from upstream.

pip will not work get-pip.py:

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/pip/
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip

If I replace the python310.zip with the one I made it works.

Collecting pip
  Using cached pip-22.3.1-py3-none-any.whl (2.1 MB)
Collecting setuptools
  Downloading setuptools-65.5.1-py3-none-any.whl (1.2 MB)
     ---------------------------------------- 1.2/1.2 MB 15.7 MB/s eta 0:00:00
Collecting wheel
  Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
Installing collected packages: wheel, setuptools, pip
  WARNING: The script wheel.exe is installed in 'C:/msys64-20220603/home/rodlie/Natron-workspace/tmp64/tmp_deploy/Natron-RB-2.5-202211171941-e0a8067-Windows-x86_64/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts pip.exe, pip3.10.exe and pip3.exe are installed in 'C:/msys64-20220603/home/rodlie/Natron-workspace/tmp64/tmp_deploy/Natron-RB-2.5-202211171941-e0a8067-Windows-x86_64/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-22.3.1 setuptools-65.5.1 wheel-0.38.4

rodlie commented 2 years ago

So, finally able to produce a RB-2.5 snapshot build. Still need to update some libs (oiio etc), and go through my build changes and commit.

I do however get two new unit test failures.

GNUVolador

GNUVolador-comp1

I assume this is because I now have a modern version of librsvg(?).

PIK

TestPIK-comp7

Don't know why this failed, will need to go through the logs.

devernay commented 2 years ago

Those failures also happen on macOS, but not on Linux. Both failures are new to 2.5.0 but were absent from 2.5.0alpha2. I'll try to give it a look, but the main difference is the Python version (3.9 on Linux, 3.10 on 2.5.0 macOS, 3.9 on 2.5.0a2 macOS)

devernay commented 2 years ago

On macOS, 2.5.0alpha2 had librsvg 2.52.4, 2.5.0 has 2.54.5

rodlie commented 2 years ago

I currently have issues with deadlocks in the unit tests. I have tried upgrading/downgrading OIIO etc, but nothing seems to work. This is an regression compared to v2.4.

rodlie commented 2 years ago

I uploaded the 2.5.0 build. Seems ok (disregarding failed tests), however the deadlocks in the unit tests are concerning and Natron may not close properly in some circumstances.

I will try to improve the new sdk sometime this week(end).

devernay commented 2 years ago

the only major difference is python 3. 2.5 probably still compiles with python 2, can you give it a test?

rodlie commented 2 years ago

I suspect this being a MSYS/MinGW issue, but will try to do a build with py2 (if possible).

bonalex01dev commented 2 years ago

Hi, I tested the 2,5 pre windows build with success: It is at least as stable as 2.4.3 Maybe a little better on the shadertoy side. (I happened to just discover that just disable/enable a shadertoy can bring it back to life) As before we get better stability with "process whole frame" (which Nuke seems to be making the norm in recent versions). Thank you for the hard work!

Some questions: Is it time to port community scripts to python 3? Is it time to advertise this new version? (pixls /discord)

devernay commented 2 years ago

OK I fixed the GNUVolador issue. That was a major issue with Python >= 3.10 (couldn't use a double-values expression for an int parameter): https://github.com/NatronGitHub/Natron/commit/54415e76e14a55a924cd64b5b26c27963a851085

Now working on the PIK failure

devernay commented 2 years ago

@bonalex01dev

Is it time to port community scripts to python 3?

Actually, it's even better to make them 2-3 compatible, using these guidelines: https://python-future.org/compatible_idioms.html

Is it time to advertise this new version? (pixls /discord)

Not until we mark 2.5.0 as a release (it's currently pre-release)

devernay commented 2 years ago

Both tests are passing now, you can start building for windows, @rodlie ! I'm updating the Linux SDK with python 3.11 before I do the linux builds

rodlie commented 2 years ago

Sorry for the delay (been sick), GNUVolador still fails here (release build).

devernay commented 2 years ago

you mean fails as before (which is just a large image difference due to a different version of librsvg I think), or fails as in the image above? I fixed the root cause for the wings not animating, so I don't see how it could fail only on windows.

rodlie commented 2 years ago

Sorry for not being specific. The test fails, but it's not the same: GNUVolador-comp1 Probably the librsvg version as you say.

Will upload the new build (still deadlocks during unit tests, will try to update to latest msys on a different instance, don't want to break anything yet).

devernay commented 2 years ago

So are we all good? I think all lights are green for the release.

Then we can focus on the migration to Qt5 (2.6) - and maybe switch the windows build to using MSVC + vcpkg

rodlie commented 2 years ago

So are we all good?

Yeah, the Windows build seems to work. Limited testing on my part (tested on a couple of workstations with some real life projects, don't see any regression compared to v2.4).

Then we can focus on the migration to Qt5 (2.6)

Ok. Will need some tweaking on Windows at least, but not anything major.

and maybe switch the windows build to using MSVC + vcpkg

That's a big maybe ;) (at least this year).

devernay commented 2 years ago

released!

bonalex01dev commented 2 years ago

Hi, to start updating the community scripts, I was able to load the "future" module in natron 2.5 with: natron-python.exe -m pip install future

but in natron 2.4.3 the same command throws an error: distutils.errors.DistutilsPlatformError: VC 6.0 is not supported by this module

My understanding is we need it on both versions to make scripts py2 py3 compatibles.

Do you have a working version of "future" for python 2.7 windows ? Or another solution ?

Thanks again for the release!

devernay commented 2 years ago

isn't future bundled with python? BTW I fixed issues with natron-plugins, they now load in 2.5

devernay commented 2 years ago

No future... I forgot to install it. We will probably release 2.4.5 and 2.5.1 with future installed

devernay commented 2 years ago

Anyway, I don't think there were any Python2-specific constructs in the community scripts. I ran "2to3 ." in the natron-plugins directory, and the only scrit that was affected is Draw/Audio_VLC/Audio_VLCExt.py which contains 2 prints (from this commit by @Windows81 https://github.com/NatronGitHub/natron-plugins/commit/ddde06429fd180f8e5540b8a51df59922d050b87 )

Windows81 commented 2 years ago

Nice to see Natron finally updating to Python 3! I'm glad none of the community plugins had real compatibility problems. I do hear that some built-in libraries available in Python 2 aren't available in 3.11 (no problems there as well). We're quite a conservative bunch!

devernay commented 2 years ago

The only issue was illegal UTF-8 characters in Python files. Fixed in https://github.com/NatronGitHub/natron-plugins/pull/18

devernay commented 2 years ago

@rodlie could you publish new 2.4.4 and 2.5.0 Windows binaries with "future" installed? I fixed the scripts here https://github.com/NatronGitHub/Natron/commit/a303f0f80532d0f775a63869cbcb9228e9c974f0

rodlie commented 2 years ago

Ok, will need to wait until I'm home (at work).

rodlie commented 2 years ago

Just uploaded v2.5.0 with future. Will start a new v2.4.4 build in a couple of hours (different computer).

rodlie commented 2 years ago

Can't install future for v2.4 using pip:

Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     |################################| 829 kB 6.6 MB/s
    ERROR: Command errored out with exit status 1:
     command: C:/msys64-20180531/home/Natron/workspace/tmp64/tmp_deploy/Natron-2.4.4-Windows-x86_64/bin/natron-python.exe -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'c:/msys64-20180531/tmp/pip-install-odxrp_/future/setup.py'"'"'; __file__='"'"'c:/msys64-20180531/tmp/pip-install-odxrp_/future/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base c:/msys64-20180531/tmp/pip-pip-egg-info-mdydis
         cwd: c:/msys64-20180531/tmp/pip-install-odxrp_/future/
    Complete output (13 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:/msys64-20180531/home/Natron/workspace/tmp64/tmp_deploy/Natron-2.4.4-Windows-x86_64/lib/python2.7/site-packages/setuptools/__init__.py", line 245, in <module>
        monkey.patch_all()
      File "C:/msys64-20180531/home/Natron/workspace/tmp64/tmp_deploy/Natron-2.4.4-Windows-x86_64/lib/python2.7/site-packages/setuptools/monkey.py", line 101, in patch_all
        patch_for_msvc_specialized_compiler()
      File "C:/msys64-20180531/home/Natron/workspace/tmp64/tmp_deploy/Natron-2.4.4-Windows-x86_64/lib/python2.7/site-packages/setuptools/monkey.py", line 164, in patch_for_msvc_specialized_compiler
        patch_func(*msvc9('find_vcvarsall'))
      File "C:/msys64-20180531/home/Natron/workspace/tmp64/tmp_deploy/Natron-2.4.4-Windows-x86_64/lib/python2.7/site-packages/setuptools/monkey.py", line 151, in patch_params
        mod = import_module(mod_name)
      File "./importlib/__init__.py", line 37, in import_module
      File "./distutils/msvc9compiler.py", line 306, in <module>
    distutils.errors.DistutilsPlatformError: VC 6.0 is not supported by this module
    ----------------------------------------

devernay commented 2 years ago

@rodlie did you try

pip uninstall setuptools
pacman -S mingw-w64-x86_x64-python-setuptools

https://stackoverflow.com/a/55554854

devernay commented 2 years ago

reopening to fix the "pip install future"

rodlie commented 2 years ago

I have mingw setuptools installed, but setuptools was installed by pip since the mingw setuptools was not in the tmp deploy folder. Should have a new build done within an hour.

NatronGitHub / Natron