Closed dumol closed 1 year ago
# Version 1.1.1 pinned since it works on both py2 and py3.
MARKUPSAFE_VERSION="1.1.1"
# Version 2.0.11 pinned since it works on both py2 and py3.
CHARSETNORMALIZER_VERSION="2.0.11"
For 3.11 we want both packages to the latest versions.
The important thing is to have them available for musl
and m1
For windows you can copy (if not already) the latest versions for pypi.org
pyopenssl can be excluded
There's a subtle (linking?) issue with the generic Linux build on Amazon 2. When building Python, you get this:
checking sqlite3.h usability... yes
checking sqlite3.h presence... yes
checking for sqlite3.h... yes
checking for sqlite3_bind_double in -lsqlite3... no
checking for sqlite3_column_decltype in -lsqlite3... no
checking for sqlite3_column_double in -lsqlite3... no
checking for sqlite3_complete in -lsqlite3... no
checking for sqlite3_enable_shared_cache in -lsqlite3... no
checking for sqlite3_progress_handler in -lsqlite3... no
checking for sqlite3_result_double in -lsqlite3... no
checking for sqlite3_set_authorizer in -lsqlite3... no
checking for sqlite3_trace_v2 in -lsqlite3... no
checking for sqlite3_trace in -lsqlite3... no
checking for sqlite3_value_double in -lsqlite3... no
checking for sqlite3_load_extension in -lsqlite3... no
checking for sqlite3_serialize in -lsqlite3... no
It should be:
checking sqlite3.h usability... yes
checking sqlite3.h presence... yes
checking for sqlite3.h... yes
checking for sqlite3_bind_double in -lsqlite3... yes
checking for sqlite3_column_decltype in -lsqlite3... yes
checking for sqlite3_column_double in -lsqlite3... yes
checking for sqlite3_complete in -lsqlite3... yes
checking for sqlite3_enable_shared_cache in -lsqlite3... yes
checking for sqlite3_progress_handler in -lsqlite3... yes
checking for sqlite3_result_double in -lsqlite3... yes
checking for sqlite3_set_authorizer in -lsqlite3... yes
checking for sqlite3_trace_v2 in -lsqlite3... yes
checking for sqlite3_value_double in -lsqlite3... yes
checking for sqlite3_load_extension in -lsqlite3... yes
checking for sqlite3_serialize in -lsqlite3... yes
as you can see in the other Linux build on Alpine 3.12.
I couldn't figure this out yet… Any ideas, @adi?
Is sqlite removed from Amazon LInux 2 OS ? As my first guess, is that somehow the Python is trying to use the OS Amazon SQLite version, and not using our latest sqlite version.
And it can't find a newer sqlite so it fails to build.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.37.2'
Mișu, I think is best to build pythia on our VMs.
It's much easier to debug.
The way we do it now with staring an Ubuntu and inside ubuntu starting an AMZ linux, is complicated and it looks like tmate doesn't work.
Tmate doesn't help when building under Docker through GHA.
Usually, in such a case, I debug in a docker container started from the same image.
Is sqlite removed from Amazon LInux 2 OS ? As my first guess, is that somehow the Python is trying to use the OS Amazon SQLite version, and not using our latest sqlite version.
You can't remove the sqlite
package from Amazon Linux 2, systemd
depends on it. Its headers are not present.
And it can't find a newer sqlite so it fails to build.
>>> import sqlite3 >>> sqlite3.sqlite_version '3.37.2'
Sorry, where is this from?
Further debugging for the errors at ./configure
shows this in build/Python-3.11.3/config.log
on Amazon Linux 2:
configure:12595: checking for LIBSQLITE3
configure:12602: $PKG_CONFIG --exists --print-errors "sqlite3 >= 3.7.15"
configure:12605: $? = 0
configure:12619: $PKG_CONFIG --exists --print-errors "sqlite3 >= 3.7.15"
configure:12622: $? = 0
configure:12668: result: yes
configure:12685: checking sqlite3.h usability
configure:12685: gcc -c -fPIC -I/home/chevah/pythia/build/python3.11-linux-x64
/include -I/include -I$(srcdir)/Modules/_sqlite conftest.c >&5
configure:12685: $? = 0
configure:12685: result: yes
configure:12685: checking sqlite3.h presence
configure:12685: gcc -E -I/home/chevah/pythia/build/python3.11-linux-x64/includ
e -I/include -I$(srcdir)/Modules/_sqlite conftest.c
configure:12685: $? = 0
configure:12685: result: yes
configure:12685: checking for sqlite3.h
configure:12685: result: yes
configure:12708: gcc -c -fPIC -I/home/chevah/pythia/build/python3.11-linux-x64
/include -I/include -I$(srcdir)/Modules/_sqlite conftest.c >&5
configure:12708: $? = 0
configure:12712: checking for sqlite3_bind_double in -lsqlite3
configure:12737: gcc -o conftest -fPIC -I/home/chevah/pythia/build/python3.11-linux-x64/include -I/include -I$(srcdir)/Modules/_sqlite -L/lib -lsqlite3 -Wl,-rpath,/home/chevah/pythia/build/python3.11-linux-x64/lib/ -L/home/chevah/pythia/build/python3.11-linux-x64/lib/ conftest.c -lsqlite3 -lpthread -ldl >&5
/home/chevah/pythia/build/python3.11-linux-x64/lib//libsqlite3.a(sqlite3.o): In function `xCeil':
sqlite3.c:(.text+0x7a9af): undefined reference to `ceil'
/home/chevah/pythia/build/python3.11-linux-x64/lib//libsqlite3.a(sqlite3.o): In function `xFloor':
sqlite3.c:(.text+0x7a9de): undefined reference to `floor'
/home/chevah/pythia/build/python3.11-linux-x64/lib//libsqlite3.a(sqlite3.o): In function `logFunc':
sqlite3.c:(.text+0x7aa7b): undefined reference to `log'
sqlite3.c:(.text+0x7aad1): undefined reference to `log'
sqlite3.c:(.text+0x7ab07): undefined reference to `log10'
sqlite3.c:(.text+0x7ab24): undefined reference to `log2'
sqlite3.c:(.text+0x7ab41): undefined reference to `log'
[…]
Which results in:
checking for stdlib extension module _sqlite3... missing
And ends with:
The necessary bits to build these optional modules were not found:
_curses _curses_panel _dbm
_gdbm _sqlite3 _tkinter
_uuid readline
The Amazon Linux 2 issue seems to be caused by something in Python 3.11 sources, as simply downgrading back to latest Python 3.10 sources fixes the SQLite-related errors. (This is on top of all the other changes in this branch, including the changes to Python's install prefix.)
Managed to figure this out after some more debugging. Which helped with further cleaning up how Python's modules find required headers.
Tmate doesn't help when building under Docker through GHA.
Correction, the tmate action doesn't help on Amazon Linux 2. But it does work on Alpine, and it also used to work on Ubuntu 18.04, even under Docker.
The changes look awesome @dumol . Great work.
@dumol can you please try to use this version as part of the chehvah/server PR ?
The diff over here looks good.
With the chevah/server PR we can see if something is misssing
Thanks
A testing version tried at https://github.com/chevah/server/pull/6230.
Scope
Use the latest Python 3.11 sources.
Changes
Updated Python version to 3.11.3.
Other updates:
pip
23.1.2setuptools
67.7.2markupsafe
2.1.2charset_normalizer
3.1.0Drive-by changes:
CPPFLAGS
.pythia.sh
from server repo.Testing
Please review changes.
Check the automated tests.
Try the uploaded testing packages.