Working with python version newer than 3.10

[jnboehm]

Thanks for putting this library together.

I wanted to use pyffi to play around a bit but unfortunately it does not appear to work with versions other than 3.10. Is there some limitation behind it or would it suffice to point pyffi to the 3.11 library? Maybe that could also be reflected in the documentation then.

If there is more work necessary, then I would be willing to look into it (though I don't have much prerequisite knowledge).

Cheers!

[soegaard]

@jnboehm Try it!

I haven't tested whether Pyffi works with 3.11 or not. Since the C API is used any change to parts of the C API, I am using, is problematic - so there is no guarantee it will work.

On the bright side, I have used only parts of the C API that seemed stable, so there is a good chance it will work.

/Jens Axel

[soegaard]

A little info of the "C API Stability".

https://docs.python.org/3/c-api/stable.html

[jnboehm]

Cool, that's good to hear! I'm actually running python3.12, but I suppose the same caveats apply there.

When I simply run raco pyffi configure it picks up the correct python interpreter, but unfortunately it seems like some things are hardcoded from the racket side. I get the following error and the repl crashes when I try to initialize pyffi:

> (initialize)
Python path configuration:
  PYTHONHOME = '/usr'
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/usr/lib/python3.10/python3.12'
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/usr'
  sys.base_exec_prefix = '/usr'
  sys.platlibdir = '/usr/lib/python3.10'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/usr'
  sys.exec_prefix = '/usr'
  sys.path = [
    '/usr/lib/python3.10/python312.zip',
    '/usr/lib/python3.10/python3.12',
    '/usr/lib/python3.10/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007611a9503740 (most recent call first):
  <no Python frame>

It looks like stdlib dir is not set correctly as well as sys.path (which I thought would be set by python itself. That may be the reason why it cannot find the fs encoding.

[soegaard]

This line looks odd to me:

stdlib dir = '/usr/lib/python3.10/python3.12'

In the terminal, if you simply write python3 which python is invoked? Use which python (for the path) and python3 -m sysconfig (for everything else) to see all about the default python installation.

If python3 works in the terminal, I think, raco pyffi configure should pick up the correct python installation. See the code for pyffi configure here:

https://github.com/soegaard/pyffi/blob/main/pyffi-lib/pyffi/configure-pyffi.rkt

[soegaard]

Apropos hardcoded, try changing python3.10 to python3.12 in python-initialization.rkt.

https://github.com/soegaard/pyffi/blob/main/pyffi-lib/pyffi/python-initialization.rkt

[jnboehm]

I managed to get pyffi to run, at least the basic functionality seems to work. Thanks for the hints. I applied the following changes:

diff --git a/pyffi-lib/pyffi/python-initialization.rkt b/pyffi-lib/pyffi/python-initialization.rkt
index 67834aa..fe5712a 100644
--- a/pyffi-lib/pyffi/python-initialization.rkt
+++ b/pyffi-lib/pyffi/python-initialization.rkt
@@ -17,7 +17,7 @@
 ;;;

 ; (define program-full-path "/Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10")
-(define program-full-path "python3.10")
+(define program-full-path "python3.12")

 (define home (get-preference 'pyffi:data (λ () #f)))
 (unless home
@@ -46,7 +46,7 @@

 (define (set-environment-variables)
   (define (decode s) (Py_DecodeLocale s #f))
-  (Py_SetProgramName (decode "python3.10"))
+  (Py_SetProgramName (decode "python3.12"))
   ; (Py_SetProgramName (decode (build-path libdir)))
   ; (Py_SetPath (Py_DecodeLocale (get-preference 'pyffi:data (λ () #f)) #f))
   (Py_SetPythonHome  (decode home)))
@@ -127,7 +127,7 @@

   (set-PyConfig-home!         config (decode home))
   ; (set-PyConfig-program_name! config (decode "python3.10"))
-  (set-PyConfig-platlibdir! config   (decode (string-append home "/" "lib/python3.10")))
+  (set-PyConfig-platlibdir! config   (decode (string-append home "/" "lib")))

   #;(let ([pythonpath (getenv "PYTHONPATH")])
     (when pythonpath

Is this something that should be extended so that it would work more generally? (One thing that surprised me is that the set-PyConfig-platlibdir! had to be changed to lib and not to lib/pythong3.12, not sure why that is the case.)

There are also some tests that fail (e.g. pyffi-tests/tests/test-string.rkt), it's probably related to requireing pyffi directly instead of the local file (as they report to be running on python3.10). I guess that could also be fixed somewhat easily.

One other detail that I noticed, which may be unrelated to the version, is that running (run* "invalid syntax") finishes without complaining, though I would have expected to get an error from that. Does that also happen with python3.10?

In any case, thank you for the pointers, that was already helpful.