Support Python 3.11 in the Python target

[cmnrd]

1455 has shown that the Python target does not work correctly with the recently released Python version 3.11. We should update the generator and/or runtime to ensure compatibility with Python 3.11.

[edwardalee]

With Python version 3.11.3, running Composition.lf results in a segmentation fault. The stack trace is here:

Process 52986 launched: '/usr/local/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python' (x86_64)
Process 52986 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
    frame #0: 0x0000000103c0c748 Python`PyDict_New + 21
Python`PyDict_New:
->  0x103c0c748 <+21>: movq   0x10(%rax), %rax
    0x103c0c74c <+25>: movl   0x2c40(%rax), %ecx
    0x103c0c752 <+31>: testl  %ecx, %ecx
    0x103c0c754 <+33>: je     0x103c0c7b8               ; <+133>
Target 0: (Python) stopped.
(lldb) thread backtrace
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
  * frame #0: 0x0000000103c0c748 Python`PyDict_New + 21
    frame #1: 0x0000000103c3c0f0 Python`PyType_Ready + 4772
    frame #2: 0x0000000100dbcfdd LinguaFrancaComposition.so`PyInit_LinguaFrancaComposition at pythontarget.c:324:9 [opt]
    frame #3: 0x00000001008bd34c Python`_imp_create_dynamic + 1350
    frame #4: 0x00000001007da291 Python`cfunction_vectorcall_FASTCALL + 85
    frame #5: 0x0000000100885630 Python`_PyEval_EvalFrameDefault + 69055
    frame #6: 0x0000000100888933 Python`_PyEval_Vector + 92
    frame #7: 0x000000010078c010 Python`object_vacall + 264
    frame #8: 0x000000010078beae Python`PyObject_CallMethodObjArgs + 226
    frame #9: 0x00000001008b9565 Python`PyImport_ImportModuleLevelObject + 2701
    frame #10: 0x000000010087d66b Python`_PyEval_EvalFrameDefault + 36346
    frame #11: 0x0000000100873ab0 Python`PyEval_EvalCode + 283
    frame #12: 0x00000001008dc2dc Python`run_eval_code_obj + 72
    frame #13: 0x00000001008dc26c Python`run_mod + 96
    frame #14: 0x00000001008dbf70 Python`pyrun_file + 133
    frame #15: 0x00000001008db9a6 Python`_PyRun_SimpleFileObject + 275
    frame #16: 0x00000001008db0b6 Python`_PyRun_AnyFileObject + 155
    frame #17: 0x00000001008fd934 Python`pymain_run_file_obj + 234
    frame #18: 0x00000001008fd161 Python`pymain_run_file + 85
    frame #19: 0x00000001008fcb2a Python`Py_RunMain + 1044
    frame #20: 0x00000001008fdd68 Python`Py_BytesMain + 42
    frame #21: 0x000000010001252e dyld`start + 462

Line 324 of python target.c is the if statement here:

    // Initialize the port_capsule type
    if (PyType_Ready(&py_port_capsule_t) < 0) {
        return NULL;
    }

Apparently, py_port_capsule_t is a new type definition for Python, and something has changed in Python w.r.t. how new types are defined.

The segfault occurs because the above call to PyType_Ready occurs before the call to Py_Initialize() (which is in py_main in the same file). I tried forcing Py_Initialize() to be called earlier, and this allowed the code to get a bit further, but it still failed.

Possibly useful resources:

• using VS Code debugger with mixed Python and C
• overview of Type Objects in Python.
• what's new in Python 11 and change logs

[lsk567]

I want to add a data point on my end. I am running Python 3.10 (same one as the one used in the macOS GitHub action) on macOS and got a segmentation fault for ActionDelay.lf. So perhaps it's not the problem with Python 3.11 but something else in the runtime.

(lfpy) shaokai@edr32731 lingua-franca % python --version
Python 3.10.12
(lfpy) shaokai@edr32731 lingua-franca % lfc -c test/Python/src/ActionDelay.lf

> Task :cli:lfc:run
Cleaning /Users/shaokai/git/lingua-franca/test/Python/bin
Cleaning /Users/shaokai/git/lingua-franca/test/Python/src-gen
lfc: info: Generating code for: file:/Users/shaokai/git/lingua-franca/test/Python/src/ActionDelay.lf
lfc: info: Generation mode: STANDALONE
lfc: info: Generating sources into: /Users/shaokai/git/lingua-franca/test/Python/src-gen/ActionDelay
Cleaning /Users/shaokai/git/lingua-franca/test/Python/include
--- Current working directory: /Users/shaokai/git/lingua-franca/test/Python/src-gen/ActionDelay/build
--- Executing command: cmake -DLF_REACTION_GRAPH_BREADTH=1 -DLF_UNTHREADED=1 -DLOG_LEVEL=2 -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/Users/shaokai/git/lingua-franca/test/Python -DCMAKE_INSTALL_BINDIR=bin -DLF_FILE_SEPARATOR="/" -DLF_SOURCE_DIRECTORY="/Users/shaokai/git/lingua-franca/test/Python/src" -DLF_PACKAGE_DIRECTORY="/Users/shaokai/git/lingua-franca/test/Python" /Users/shaokai/git/lingua-franca/test/Python/src-gen/ActionDelay
-- The C compiler identification is AppleClang 13.0.0.13000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Including sources for unthreaded runtime.
-- Including the following sources: tag.c, port.c, mixed_radix.c, reactor_common.c, lf_token.c, environment.c, reactor.c, vector.c, pqueue.c, util.c, semaphore.c, hashset.c, hashset_itr.c, modes.c, lf_unix_clock_support.c, lf_unix_syscall_support.c, lf_linux_support.c, lf_macos_support.c, lf_windows_support.c, lf_nrf52_support.c, lf_zephyr_support.c

(... compiling ...)

[ 97%] Building C object CMakeFiles/LinguaFrancaActionDelay.dir/ActionDelay.c.o
[100%] Linking C shared module /Users/shaokai/git/lingua-franca/test/Python/src-gen/ActionDelay/LinguaFrancaActionDelay.so
[100%] Built target LinguaFrancaActionDelay
Install the project...
-- Install configuration: "Release"
lfc: info: SUCCESS: Compiling generated code for ActionDelay finished with no errors.
lfc: info: Compiled binary is in /Users/shaokai/git/lingua-franca/test/Python/bin
lfc: info:
#####################################
To run the generated program, use:

    python3 /Users/shaokai/git/lingua-franca/test/Python/src-gen/ActionDelay/ActionDelay.py

#####################################

lfc: info: Code generation finished.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 11s
20 actionable tasks: 1 executed, 19 up-to-date
(lfpy) shaokai@edr32731 lingua-franca % python3 --version
Python 3.10.12
(lfpy) shaokai@edr32731 lingua-franca % pip3 show LinguaFrancaBase
Name: LinguaFrancaBase
Version: 1.0.3
Summary: Useful notations and helper functions used in Lingua Franca programs.
Home-page: https://github.com/icyphy/lingua-franca
Author: Soroush Bateni
Author-email: soroosh129@gmail.com
License: UNKNOWN
Location: /Users/shaokai/anaconda3/envs/lfpy/lib/python3.10/site-packages
Requires:
Required-by:
(lfpy) shaokai@edr32731 lingua-franca % python3 /Users/shaokai/git/lingua-franca/test/Python/src-gen/ActionDelay/ActionDelay.py
zsh: segmentation fault  python3
(lfpy) shaokai@edr32731 lingua-franca %

[edwardalee]

I am not able to reproduce this. Note that you are using the command python to check the version but python3 to run the program. Are you sure python3 is in fact Python 3.10?

[lsk567]

Yes, I am inside a Conda environment lfpy with Python 3.10 installed. So both commands return the same version. I am not sure why this is not reproducible...

(lfpy) shaokai@edr32731 lingua-franca % python3 /Users/shaokai/git/lingua-franca/test/Python/src-gen/ActionDelay/ActionDelay.py
zsh: segmentation fault  python3
(lfpy) shaokai@edr32731 lingua-franca % ls
CHANGELOG.md      LICENSE           bin               build.gradle      cli               core              gradle.properties gradlew.bat       settings.gradle   util
CONTRIBUTING.md   README.md         build             buildSrc          codecov.yml       gradle            gradlew           lsp               test              workspace
(lfpy) shaokai@edr32731 lingua-franca % python3 --version
Python 3.10.12
(lfpy) shaokai@edr32731 lingua-franca % python --version
Python 3.10.12
(lfpy) shaokai@edr32731 lingua-franca %

[edwardalee]

I've created a branch of both lingua-franca and reactor-c called python-11 that has a start on this. It gets passed the segfault, but fails to import the module. The reason is unclear.

[edwardalee]

The branch python-11 (lingua-franca and reactor-c) now passes a smoke test. It requires Python 3.11 and will not work with Python 3.10. How can we run all the tests using Python 3.11? If they pass, this offers a solution for the next release, which would require Python 3.11.

[lhstrh]

While I don't think we need this for the release, it would be fine to bump our requirement for Python to 3.11. I can adjust CI in this branch to that effect...

[edwardalee]

To further document this issue, on my machine, it seems that cmake was finding a different Python from the one in my path. This is probably also happening to others, which could account for the segfaults that @lsk567 is seeing. The output from lfc includes this line:

-- Found Python: /usr/local/Frameworks/Python.framework/Versions/3.10/bin/python3.10 (found suitable version "3.10.11", required range is "3.7.0...<3.12.0") found components: Interpreter Development Development.Module Development.Embed 

This Python is not even in my PATH, so I have no idea how or why cmake would choose it. The version of python3 in my path is 3.11. I managed to get rid of the segfaults even with the mismatch, but couldn't import the generated extension module. The error message was this:

SystemError: initialization of LinguaFrancaActionDelay did not return an extension module

The fix for this was to change the code generator to generate a cmake file that requires Python 3.11. I'm not sure whether there is a better way to ensure that the version of Python used to generate the code is the same as the version in the user's path.

[lhstrh]

Matching the Python version may be more flexible, but also more difficult to maintain and likely to cause issues down the road...

[lsk567]

Thanks for taking a closer look, @edwardalee. This is exactly the issue. On my machine, cmake finds Python 3.7, which is not the version I use in my conda environment.

-- Found Python: /Users/shaokai/anaconda3/bin/python3.7 (found suitable version "3.7.16", required range is "3.7
.0...<3.11.0") found components: Interpreter Development Development.Module Development.Embed
-- Configuring done (20.3s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/shaokai/git/lingua-franca/test/Python/src-gen/ActionDelay/build
--- Current working directory: /Users/shaokai/git/lingua-franca/test/Python/src-gen/ActionDelay/build
--- Executing command: cmake --build . --target install --parallel 16 --config Release

Updating the requirement to Python 3.10 seems to solve the issue.

-- Applying preprocessor definitions...
-- Found Python: /Users/shaokai/anaconda3/envs/lfpy/bin/python3.10 (found suitable version "3.10.12", required r
ange is "3.10.0...<3.11.0") found components: Interpreter Development Development.Module Development.Embed
-- Configuring done (16.2s)
-- Generating done (0.0s)

The corollary seems to be that we should raise the lower bound to be above Python 3.7 at least.

Update: /Users/shaokai/anaconda3/envs/lfpy/bin/python3.10 also segfaulted.

[cmnrd]

According to the cmake documentation of FindPython (in particular the hints section), cmake by default prefers the newest version that it can find. On MacOs, it also prefers framework versions of python over unix-style packages. This explains why cmake chose the framework python over the python in the PATH in Edward's post. The behavior can be changed globally by setting CMAKE_FIND_FRAMEWORK to LAST. It can also be changed just for Python by setting Python_FIND_FRAMEWORK to LAST.

I am not sure why cmake chose Python 3.7 in Shaokai's post. We had a debugging session today, and we were not able to reproduce this. cmake consistently chose Python 3.10 for various settings.

We made three important observations today:

1. The segmentation fault in Shaokai's setup only occurs when using the anaconda Python. I did not find reliable sources, but it seems that Anaconda ships a custom Python interpreter that is a modified version of CPython. Perhaps there is some incompatibility with Anaconda's interpreter. When using the homebrew or framework version of Python 3.10, everything works fine for Shaokai.
2. The segmentation faults that we have been seeing appear simply to be caused by compiling and linking the C files with a different version of Python than the one used for execution. Cmake does not prefer the Python interpreter on PATH and there is no reliable way to enforce this (as the interpreter on path might not meet the Python version requirements). This problem is orthogonal to the one reported initially in this issue.
3. HelloWolrld.lf actually runs fine with Python3.11 on my Linux machine (using master). Unfortunately, I did not document the precise error that we observed earlier when I first opened the issue and the CI log is not available anymore. We should run the full test suite with Python 3.11 again and see where it fails. The segmentation errors that we have discussed, seem to be an unrelated issue.

I think we should try to resolve the discrepancy between the default python interpreter of the system and the one selected by cmake. Since it is difficult to influence cmakes choice in a portable way, I think we should push the user to use the right interpreter. We could simply parse cmake's output and extract the path to the interpreter that it selected. On my machine cmake prints Found Python: /usr/bin/python3.11. So instead of printing

To run the generated program, use: 
    python3 /home/cmenard/projects/lf/lingua-franca/test/Python/src-gen/HelloWorld/HelloWorld.py

we could print:

To run the generated program, use:
     /usr/bin/python3.11 /home/cmenard/projects/lf/lingua-franca/test/Python/src-gen/HelloWorld/HelloWorld.py

We could even take this one step further and generate a bash (or batch on windows) script in bin/ that simply runs the above command.

A more reliable way than parsing the output to get the Python interpreter path could be to let cmake write the path to a file using file(WRITE "python_interpreter.txt" "${Python_EXECUTABLE}").

UPDATE: Actually we could let cmake generate the launchscipt and completely avoid the feedback from cmake to lfc.

[edwardalee]

Generating a launch script in bin seems like just the right thing to do. This would make launching programs consistent and it encodes in a more permanent way the output from lfc.

[lhstrh]

Looks like this was fixed in #1902.

[edwardalee]

But #1902 is not merged and has test failures.

[lhstrh]

:hand_over_mouth: I missed that.