Package Python 3.11.

[dumol]

Scope

Use the latest Python 3.11 sources.

Changes

Updated Python version to 3.11.3.

Other updates:

• SQLite 3.41.2
• pip 23.1.2
• setuptools 67.7.2
• markupsafe 2.1.2
• charset_normalizer 3.1.0

Drive-by changes:

• Cleared up the use of CPPFLAGS.
• Slightly improved Python build script.
• Updated pythia.sh from server repo.

Testing

Please review changes.

Check the automated tests.

Try the uploaded testing packages.

[adiroiban]

# Version 1.1.1 pinned since it works on both py2 and py3.
MARKUPSAFE_VERSION="1.1.1"
# Version 2.0.11 pinned since it works on both py2 and py3.
CHARSETNORMALIZER_VERSION="2.0.11"

For 3.11 we want both packages to the latest versions. The important thing is to have them available for musl and m1

For windows you can copy (if not already) the latest versions for pypi.org

---

pyopenssl can be excluded

[dumol]

There's a subtle (linking?) issue with the generic Linux build on Amazon 2. When building Python, you get this:

  checking sqlite3.h usability... yes
  checking sqlite3.h presence... yes
  checking for sqlite3.h... yes
  checking for sqlite3_bind_double in -lsqlite3... no
  checking for sqlite3_column_decltype in -lsqlite3... no
  checking for sqlite3_column_double in -lsqlite3... no
  checking for sqlite3_complete in -lsqlite3... no
  checking for sqlite3_enable_shared_cache in -lsqlite3... no
  checking for sqlite3_progress_handler in -lsqlite3... no
  checking for sqlite3_result_double in -lsqlite3... no
  checking for sqlite3_set_authorizer in -lsqlite3... no
  checking for sqlite3_trace_v2 in -lsqlite3... no
  checking for sqlite3_trace in -lsqlite3... no
  checking for sqlite3_value_double in -lsqlite3... no
  checking for sqlite3_load_extension in -lsqlite3... no
  checking for sqlite3_serialize in -lsqlite3... no

It should be:

  checking sqlite3.h usability... yes
  checking sqlite3.h presence... yes
  checking for sqlite3.h... yes
  checking for sqlite3_bind_double in -lsqlite3... yes
  checking for sqlite3_column_decltype in -lsqlite3... yes
  checking for sqlite3_column_double in -lsqlite3... yes
  checking for sqlite3_complete in -lsqlite3... yes
  checking for sqlite3_enable_shared_cache in -lsqlite3... yes
  checking for sqlite3_progress_handler in -lsqlite3... yes
  checking for sqlite3_result_double in -lsqlite3... yes
  checking for sqlite3_set_authorizer in -lsqlite3... yes
  checking for sqlite3_trace_v2 in -lsqlite3... yes
  checking for sqlite3_value_double in -lsqlite3... yes
  checking for sqlite3_load_extension in -lsqlite3... yes
  checking for sqlite3_serialize in -lsqlite3... yes

as you can see in the other Linux build on Alpine 3.12.

I couldn't figure this out yet… Any ideas, @adi?

[adiroiban]

Is sqlite removed from Amazon LInux 2 OS ? As my first guess, is that somehow the Python is trying to use the OS Amazon SQLite version, and not using our latest sqlite version.

And it can't find a newer sqlite so it fails to build.

---

>>> import sqlite3
>>> sqlite3.sqlite_version
'3.37.2'

[adiroiban]

Mișu, I think is best to build pythia on our VMs.

It's much easier to debug.

The way we do it now with staring an Ubuntu and inside ubuntu starting an AMZ linux, is complicated and it looks like tmate doesn't work.

[dumol]

Tmate doesn't help when building under Docker through GHA.

Usually, in such a case, I debug in a docker container started from the same image.

[dumol]

Is sqlite removed from Amazon LInux 2 OS ? As my first guess, is that somehow the Python is trying to use the OS Amazon SQLite version, and not using our latest sqlite version.

You can't remove the sqlite package from Amazon Linux 2, systemd depends on it. Its headers are not present.

And it can't find a newer sqlite so it fails to build.

>>> import sqlite3
>>> sqlite3.sqlite_version
'3.37.2'

Sorry, where is this from?

[dumol]

Further debugging for the errors at ./configure shows this in build/Python-3.11.3/config.log on Amazon Linux 2:

configure:12595: checking for LIBSQLITE3
configure:12602: $PKG_CONFIG --exists --print-errors "sqlite3 >= 3.7.15"
configure:12605: $? = 0
configure:12619: $PKG_CONFIG --exists --print-errors "sqlite3 >= 3.7.15"
configure:12622: $? = 0
configure:12668: result: yes
configure:12685: checking sqlite3.h usability
configure:12685: gcc -c  -fPIC  -I/home/chevah/pythia/build/python3.11-linux-x64
/include -I/include   -I$(srcdir)/Modules/_sqlite conftest.c >&5
configure:12685: $? = 0
configure:12685: result: yes
configure:12685: checking sqlite3.h presence
configure:12685: gcc -E  -I/home/chevah/pythia/build/python3.11-linux-x64/includ
e -I/include   -I$(srcdir)/Modules/_sqlite conftest.c
configure:12685: $? = 0
configure:12685: result: yes
configure:12685: checking for sqlite3.h
configure:12685: result: yes
configure:12708: gcc -c  -fPIC  -I/home/chevah/pythia/build/python3.11-linux-x64
/include -I/include   -I$(srcdir)/Modules/_sqlite conftest.c >&5
configure:12708: $? = 0
configure:12712: checking for sqlite3_bind_double in -lsqlite3
configure:12737: gcc -o conftest  -fPIC  -I/home/chevah/pythia/build/python3.11-linux-x64/include -I/include   -I$(srcdir)/Modules/_sqlite -L/lib -lsqlite3   -Wl,-rpath,/home/chevah/pythia/build/python3.11-linux-x64/lib/ -L/home/chevah/pythia/build/python3.11-linux-x64/lib/  conftest.c -lsqlite3  -lpthread -ldl  >&5
/home/chevah/pythia/build/python3.11-linux-x64/lib//libsqlite3.a(sqlite3.o): In function `xCeil':
sqlite3.c:(.text+0x7a9af): undefined reference to `ceil'
/home/chevah/pythia/build/python3.11-linux-x64/lib//libsqlite3.a(sqlite3.o): In function `xFloor':
sqlite3.c:(.text+0x7a9de): undefined reference to `floor'
/home/chevah/pythia/build/python3.11-linux-x64/lib//libsqlite3.a(sqlite3.o): In function `logFunc':
sqlite3.c:(.text+0x7aa7b): undefined reference to `log'
sqlite3.c:(.text+0x7aad1): undefined reference to `log'
sqlite3.c:(.text+0x7ab07): undefined reference to `log10'
sqlite3.c:(.text+0x7ab24): undefined reference to `log2'
sqlite3.c:(.text+0x7ab41): undefined reference to `log'
[…]

Which results in:

checking for stdlib extension module _sqlite3... missing

And ends with:

The necessary bits to build these optional modules were not found:
_curses               _curses_panel         _dbm
_gdbm                 _sqlite3              _tkinter
_uuid                 readline

[dumol]

The Amazon Linux 2 issue seems to be caused by something in Python 3.11 sources, as simply downgrading back to latest Python 3.10 sources fixes the SQLite-related errors. (This is on top of all the other changes in this branch, including the changes to Python's install prefix.)

[dumol]

Managed to figure this out after some more debugging. Which helped with further cleaning up how Python's modules find required headers.

[dumol]

Tmate doesn't help when building under Docker through GHA.

Correction, the tmate action doesn't help on Amazon Linux 2. But it does work on Alpine, and it also used to work on Ubuntu 18.04, even under Docker.

[adiroiban]

The changes look awesome @dumol . Great work.

@dumol can you please try to use this version as part of the chehvah/server PR ?

The diff over here looks good.

With the chevah/server PR we can see if something is misssing

Thanks

[dumol]

A testing version tried at https://github.com/chevah/server/pull/6230.