Project updates for python 3.12 compatibility

[JacobCallahan]

SafeConfigParser has been deprecated since Python 3.2 and was removed in Python 3.12 entirely. Per the release notes [1] it is recommended to use the ConfigParser class now.

Additionally, the readfp method has been removed in favor of read_file.

Unbind the upper limit of Cython dependency and rebuild with Cython 3.0.5.

Update the vendored version of libssh2.

[1] - https://docs.python.org/3.12/whatsnew/3.12.html#configparser

[JacobCallahan]

This should resolve the first (and hopefully last) error we're seeing for #193

[JacobCallahan]

Also, would it be possible to add python 3.11 and 3.12 to the circleci runs?

[JacobCallahan]

CI failure related to https://github.com/pypa/setuptools/issues/2497 It looks like we'll need to supply a conformant version to build correctly.

[ogajduse]

@pkittenis Are you still around? Could we get your attention on this PR? ssh2-python users are not able to use the library with the newest Python unless a patch is made.

[JacobCallahan]

One option here for moving away from versioneer is what @kdart-brt did in his fork. https://github.com/pycopia/ssh2-python3/commit/ea47675f1977381b4ffd86fd95cf8e7c985f3acc

[pbrezina]

@enkore @pkittenis May I kindly ask you to review and merge this pull request?

[pbrezina]

@enkore @pkittenis If the project is dead, I would like to talk to you about giving me access to the organization. I don't have time to fully maintain it, but I can keep it functional and I would try to find some contributors. This is a great project, it would be a pity if we can't keep improving it.

[JacobCallahan]

Steps forward in latest commit, but still issues to resolve. It now builds locally in python 3.12.0

$ pip install .
Processing /home/jake/Programming/ssh2-python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: ssh2-python
  Building wheel for ssh2-python (pyproject.toml) ... done
  Created wheel for ssh2-python: filename=ssh2_python-1.0.0+2.g75176e3.dirty-cp312-cp312-linux_x86_64.whl size=786650 sha256=f130033efc5a35c3e46382234f3033cc5b4f96187f7a5647257b6a9467c8f0c1
  Stored in directory: /home/jake/.cache/pip/wheels/2c/19/89/55dc2bd0f7a333b550bcc8bbe940c10f7bde79d2959852e6de
Successfully built ssh2-python
Installing collected packages: ssh2-python
Successfully installed ssh2-python-1.0.0+2.g75176e3.dirty

However, there appears to be an issue still with the package itself.

In [1]: from ssh2 import channel
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[1], line 1
----> 1 from ssh2 import channel

File ~/Programming/ssh2-python/ssh2/channel.pyx:1, in init ssh2.channel()
----> 1 # This file is part of ssh2-python.
      2 # Copyright (C) 2017-2020 Panos Kittenis
      3 #

File ~/Programming/ssh2-python/ssh2/session.pyx:1, in init ssh2.session()
----> 1 # This file is part of ssh2-python.
      2 # Copyright (C) 2017-2020 Panos Kittenis
      3 #

ValueError: builtins.bool size changed, may indicate binary incompatibility. Expected 32 from C header, got 24 from PyObject

Additionally, the cythonize compatibility layer doesn't play nicely with CI.

[enkore]

@pbrezina @JacobCallahan I can only push to non-master branches in this org.

[JacobCallahan]

@enkore do you know of a way to contact @pkittenis? Thanks!

[enkore]

I don't

[JacobCallahan]

@enkore bah, ok. Any help with this would definitely be appreciated!

[enkore]

I've sent a mail to the mail addresses listed on Github, the website and PyPI (they're all different)

[enkore]

Regarding the issue here https://github.com/ParallelSSH/ssh2-python/pull/194#issuecomment-1806530946

The .c files need to be re-cythonized with Cython >= 0.29.34 (https://github.com/cython/cython/commit/43ce5588fb97fba407e1af04195c01b4d710453c). Cython 3 is the future of course, but that migration might be a bit more involved (mostly for setup.py, I started doing this for ssh-python)

[enkore]

I created a next branch so we can apply patches for the time being; I've done the same over at ssh-python (https://github.com/ParallelSSH/ssh-python/pull/78)

[JacobCallahan]

@enkore the latest commit is working locally in 3.12 and passing CI in 3.8 and 3.10.

Constrained the Cython version to be less than 3.0, but greater than
0.29.34.
Updated the setup.py to account for the updated cythonize parameters.

However, the appveyor build is failing with the incorrect version error.

circleci: 1.0.0+4 appveyor: 1.0.0-5

[enkore]

The cause for the appveyor build failure is that there is an extra script involved in the build to generate the version number (https://github.com/ParallelSSH/ssh2-python/blob/master/ci/appveyor/fix_version.py), which doesn't generate a PEP440 compliant version number, which is rejected by newer setuptools. I made this change over in ssh-python to work around that: https://github.com/ParallelSSH/ssh-python/pull/78/commits/3a7b14e57c5c10b6fed24aa87d66ceca4109b7e1

---

Really need AI to read that one for me :)

[JacobCallahan]

@enkore ahh yeah i was digging in and noticed that, but took a bit of a different route https://github.com/ParallelSSH/ssh2-python/pull/194/files#diff-2fe696be428908583b0de2fe7cc8c2db9089e5ca32f85ce3ab18ccad2dbca84fR8-R9

edit: And Appveyor is now green!

[JacobCallahan]

@enkore another update: we're now running into issues with the base test class setup trying to perform a handshake for the session. None of the key files have been touched in 6 years, and I'm able to reproduce these failures locally.

Another note: I update my local Cython to 3.0.5 and rebuilt. Everything built and installed as now with the versions pinned in this PR, so we are likely good to unbound the upper limit for Cython. (tests still failing though)

[JacobCallahan]

@enkore would it be useful to update the vendored libssh2? I see it's been about 1.5 years since the last update.

[pletnes]

Excited to see some activity here, this project is useful to me. Is there work being done on windows binary builds, which are missing from some releases? Is there anything I could do to help?

[JacobCallahan]

@pletnes if we can get it released via this project, it should be able to use the existing build pipeline to deliver windows binaries. If you'd like to test out the changes here, you can try pulling my fork locally and pip install from there. You shouldn't need to do anything special since the cython-compiled pieces have already been included.

[enkore]

Regarding how we can continue with the projects I created a dedicated discussion ticket here: https://github.com/ParallelSSH/parallel-ssh/issues/386

[JacobCallahan]

Huge scope expansion with the latest two commits. I've both updated the latest compiled filed with Cython 3.0.5 as well as updated the vendored libssh2 for this repo.

However, even with these updates, we're still seeing the issues with key exchanges.

Locally, password-based auth workflows are working flawlessly as exercised by my own project's tests.

[enkore]

Thank you very much! I'll merge this as-is, the KexFailure only popped up after switching the base image, so it's likely an artifact of that (might be an issue with the default settings of openssh-server in whatever distro the CI image is based on). I'll look into it.

[Red-M]

That KexFailure is due to ssh-rsa keys being depreciated in openssh and thus being made by default, a disabled key exchange for openssh-server.

[enkore]

Yes, looks like libssh2 is being built without support for the newer rsa-sha2 schemes:

>>> ssh2.session.Session().supported_algs(ssh2.session.LIBSSH2_METHOD_HOSTKEY)
['ecdsa-sha2-nistp256', 'ecdsa-sha2-nistp384', 'ecdsa-sha2-nistp521', 'ecdsa-sha2-nistp256-cert-v01@openssh.com',
 'ecdsa-sha2-nistp384-cert-v01@openssh.com', 'ecdsa-sha2-nistp521-cert-v01@openssh.com',
  'ssh-ed25519', 'ssh-rsa', 'ssh-dss']

Which is kinda strange because both the SHA1 and SHA2 variants should either all be present or entirely absent in the OpenSSL backend:

#ifdef OPENSSL_NO_RSA
# define LIBSSH2_RSA 0
# define LIBSSH2_RSA_SHA1 0
# define LIBSSH2_RSA_SHA2 0
#else
# define LIBSSH2_RSA 1
# define LIBSSH2_RSA_SHA1 1
# define LIBSSH2_RSA_SHA2 1
#endif

[enkore]

Oh so you see we have two nested copies of the libssh2 source tree and end up building the older, inner one. Whoopsie.

[pkittenis]

Should also update the https://github.com/ParallelSSH/ssh2-python/blob/master/ci/docker/manylinux/libssh2-1.10.0.tar.gz source file (used for manylinux builds) if updating libssh2 sources.

The build failures look related to which libssh2 is being used for building on the CI. Once those are resolved can be merged.

[JacobCallahan]

@pkittenis is this something you're wanting to take on, or would you like me to? If the latter, can you provide some documentation of what you'd like done? Ideally, we could use some project documentation for how to handle situations like this in the future. Thanks!