Closed dumol closed 1 year ago
Anything blocking this?
I am commenting here so that I don't forget
Here are the missing extra binary wheels.
For py3.8 on Linux, I have updated some pre-compiled versions , but we need them for Alpine.
We might just remove support for alpine for the initial py3, and later we can look into adding these packages
Saved ./bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl Saved ./cryptography-40.0.1-cp36-abi3-manylinux_2_28_x86_64.whl Saved ./cffi-1.15.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Saved ./PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
[…]
Saved ./bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl
bcrypt
4.0.1 is already present on both Linux builds.
Saved ./cryptography-40.0.1-cp36-abi3-manylinux_2_28_x86_64.whl
Updated cryptography
to newly-released version 40.0.1 through https://github.com/chevah/pythia/pull/39/commits/9eb343cb2afb961a24d6646f4c4e14423b826815.
Saved ./cffi-1.15.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cffi
1.15.1 already present on all platforms, including Alpine. Now also checked through https://github.com/chevah/pythia/pull/39/commits/ef91078c5e286cca3af23fe7c8b986f49b1deb1a.
Saved ./PyNaCl-1.5.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
PyNaCl is now built and checked for through https://github.com/chevah/pythia/pull/39/commits/7bc9f35f2204dd9c94561561fedf613f7cb974e0.
If you want a Python 3.8.x release with everything but the Python 3.10 update, let me know, I can do such a release independently of this update to Python 3.10.
I think that python 3.8 is ok.
But is important to make sure that for 3.10 we have the binary packages.
Another binary package is https://pypi.org/project/MarkupSafe/1.1.1/#files
I will continue to add the packages here
Another binary package is https://pypi.org/project/charset-normalizer/2.0.11/
Note that for now we use 2.0.11 since it works on both py2 and py3.
also on py3.8 pywin32 was missing
Basically, for py3 on both linux and Windows, we should look into our existing pypi.chevah.com files and review any file that ends with:
Now, our pypi.chevah.com already has some binary package, since I used this as a quick fix, but for the long term, these packages should be provided by pythia.
Another thing.
For now we need --use-deprecated=legacy-resolver
from pip, so this most probably means that we can only use pip 21.0
on py3.8 there is another issue on Windows.
Somehow, the site packates are not autoamtically enabled.
I can see the lib/python38._pth
which has the site.main()
disabled.
I can patch as part of our deps system, but it should be enabled in pythia by defautl.
$ cat build-py3/lib/python38._pth
python38.zip
.
# Uncomment to run site.main() automatically
#import site
lib\site-packages
Another thing that should be done for py3.10
we no longer need the -3
check
diff --git a/pythia.sh b/pythia.sh
index baa9dd52e..a3ad6a61a 100755
--- a/pythia.sh
+++ b/pythia.sh
@@ -106,7 +106,7 @@ check_source_folder() {
# Called to trigger the entry point in the virtual environment.
# Can be overwritten in pythia.conf
execute_venv() {
- ${PYTHON_BIN} $PYTHON3_CHECK -c 'from paver.tasks import main; main()' "$@"
+ ${PYTHON_BIN} -c 'from paver.tasks import main; main()' "$@"
}
@@ -905,15 +905,6 @@ if [ "$COMMAND" == "deps" ] ; then
install_base_deps
fi
-case $COMMAND in
- test_ci|test_py3)
- PYTHON3_CHECK='-3'
- ;;
- *)
- PYTHON3_CHECK=''
- ;;
-esac
-
Another thing. For now we need
--use-deprecated=legacy-resolver
from pip, so this most probably means that we can only use pip 21.0
Am I missing something? Documentation for pip 23.0.1 still shows this parameter as present: https://pip.pypa.io/en/stable/development/release-process/#use-deprecated.
Am I missing something? Documentation for pip 23.0.1 still shows this parameter as present: https://pip.pypa.io/en/stable/development/release-process/#use-deprecated.
True. it look like this feature was not removed .. yet It still works with 23.0.1 So all good for pip.
Regarding the python wheel binary packages.
It's ok to have them here.
The only hard requirement is for musl/alpine.
For windows and generic linux, I can find them already pre-compiled on the official PyPI site.
Also, it would help a lot to just run 64bit on Windows, since then we will no longer have to execute in pythia.sh the systeminfo.exe
helper... it just slows down the whole thing
on py3.8 there is another issue on Windows.
Somehow, the site packates are not autoamtically enabled.
Is this solved now with your latest changes?
Thanks for all the fixes! Is there anything left to improve? I'm a bit dizzy after checking all the commits done through suggestions above… :-]
Also, as discussed on Friday on #chevah, what Python modules should I remove from Pythia? I see charset_normalizer
is only available as a binary wheel. Other stuff?
You mentioned setproctitle
and MarkupSafe
, but the former is built from source on platforms other than Windows, while the latter is built on Alpine.
There is also cryptography
which is not built anywhere lately, but I guess you want this included, so I'm not sure where to draw the line…
On a second thought, if we don't include cryptography
, then it can be installed and packaged with SFTPPlus as needed. Which would save us from re-releasing Pythia whenever there is a cryptography
security update… Maybe useful after all?
Another thing that should be done for py3.10
we no longer need the
-3
check
Done through https://github.com/chevah/pythia/pull/39/commits/f2a18c5e583a25d41a75d167707bfa27bfd6bb01.
Went through all the comments once more, let me know if I missed something before approving this. Thanks!
For the record, I've also tried building the generic Linux package on Amazon Linux 1 (aka 2018.03), which has the same glibc version as RHEL 7. And which has gcc 7.2 available as an extra package.
The package builds with minimal changes, but it has the same issues as on CentOS 7, it links against /lib64/libfreebl3.so
, which belongs to an irremovable system package.
I am leaving a comment here, but most probably needs to be done in a separate ticket.
For the Windows version, besides the lib/python310.zip we also need to distribute the source code.
Without the source code, the python works, but debugging is hard.
So for Windows, besides the current embeddable version, we need to find a way to also get the source code avaiable inside the build folder
So for Windows, besides the current embeddable version, we need to find a way to also get the source code avaiable inside the build folder
Sure, no problem, I'll fill in a new ticket… But what sources you want included? The win package of python-package only has some include files in /python2.7-win-x64/lib/include/
… You want more? All the source code of Python? Only some parts of it?
What kind of debugging you have in mind? Debugging Python binaries?
For the record, merged into master as discussed on #chevah last week on Friday.
Scope
Update Python.
Changes
Use latest Python 3.10. (Python 3.11 builds fail because of https://github.com/python/cpython/issues/101668.)
The
libedit
patch was removed, as it's not worth porting it to apply to 3.10 sources only for building on macOS…Drive-by changes:
-latest
(currently Windows Server 2022 and macOS 12).cryptography
to newly-released version 40.0.1.cffi
too.MarkupSafe
,charset-normalizer
, andpywin32
(the latter only on Windows).Testing
Review changes and the automated test results.
Try the resulting test packages, e.g. https://bin.chevah.com:20443/testing/3.10.11.5cdf347/