REPL editing and history.

[dumol]

That would require the readline module, which we can only be build against libedit because of licence issues.

When we updated from Python 3.8.x to 3.10.x, the custom patch we used wasn't updated. Instead, we stopped building readline.

However, the libedit support was not completely removed, I've left the starting bits and a link to an updated patch in place: https://github.com/chevah/pythia/blob/master/build.conf#L15.

Try to get this back. At least for Linux…

[dumol]

While at it, maybe also evaluate possible alternatives, e.g. https://github.com/prompt-toolkit/ptpython.

[adiroiban]

Note that we don't really need read–eval–print loop (REPL), but rather support for terminal emulation.

But I think that this is provided by https://github.com/prompt-toolkit/python-prompt-toolkit

So, before looking into libedit, let me try python-prompt-toolkit ... maybe we can get something working with python only code.

thanks

[dumol]

While updating to Python 3.13 in #58, I've noticed readline got built on macOS by default. It links against system libedit libs, so it's portable enough to be left enabled.

I've gave Alpine Linux and generic Linux a quick try to re-enable building against libedit, but it's not that straightforward, it looks like a patch like https://github.com/chevah/pythia/blob/09c128154d23feb6b1a7cb5a8d79efd938769e69/src/Python/readline_libedit.patch is still needed. A starting diff for Pythia:

diff --git a/.github/workflows/docker.yaml b/.github/workflows/docker.yaml
index 75db0dc..97b62fc 100644
--- a/.github/workflows/docker.yaml
+++ b/.github/workflows/docker.yaml
@@ -35,7 +35,7 @@ jobs:
       if: startsWith(matrix.container, 'alpine')
       run: |
         apk upgrade -U
-        apk add git curl bash gcc make m4 patch musl-dev linux-headers lddtree shadow openssh-client file unzip perl g++ musl-locales dejagnu
+        apk add git curl bash gcc make m4 patch musl-dev linux-headers lddtree shadow openssh-client file unzip perl g++ musl-locales dejagnu automake libtool ncurses-dev
         apk del util-linux-dev
         curl --output /usr/local/bin/paxctl https://bin.chevah.com:20443/third-party-stuff/alpine/paxctl-3.12
         chmod +x /usr/local/bin/paxctl
@@ -44,7 +44,7 @@ jobs:
       if: startsWith(matrix.container, 'amazonlinux')
       run: |
         yum -y upgrade
-        yum -y install git-core gcc make m4 patch tar unzip perl perl-Test-Simple perl-IPC-Cmd xz gcc-c++ dejagnu
+        yum -y install git-core gcc make m4 patch tar unzip perl perl-Test-Simple perl-IPC-Cmd xz gcc-c++ dejagnu automake libtool ncurses-devel
         # To avoid linking against libdb and gdmb libraries on Amazon Linux 2.
         # Can't simply uninstall libdb-devel and gdmb-devel, they are required by perl-IPC-Cmd.
         rm -v /usr/include/libdb/db.h
@@ -55,7 +55,7 @@ jobs:
       run: |
         apt update
         apt --yes dist-upgrade
-        apt --yes install wget curl gcc make m4 patch openssh-client unzip git libtest-simple-perl xz-utils g++ dejagnu
+        apt --yes install wget curl gcc make m4 patch openssh-client unzip git libtest-simple-perl xz-utils g++ dejagnu automake libtool libncurses5-dev

     - name: Clone repo independently
       run: |
diff --git a/build.conf b/build.conf
index 43cb946..dd828e2 100644
--- a/build.conf
+++ b/build.conf
@@ -22,7 +22,7 @@ XZ_VERSION="5.6.2"
 # If there's a need to reenable this, our libedit patch for Python 3.9 was
 # https://github.com/chevah/pythia/pull/5/commits/09c128154d23feb6b1a7cb5a8d79.
 # A newer patch is available at https://github.com/python/cpython/issues/57710.
-LIBEDIT_VERSION="20170329-3.1"
+LIBEDIT_VERSION="20240517-3.1"
 # Our OpenSSL libs are only used for Python's "ssl" module lately.
 OPENSSL_VERSION="3.0.14"
 # Use the version of the "sqlite-autoconf-VERSION.tar.gz" upstream download.
diff --git a/os_quirks.sh b/os_quirks.sh
index bba1955..a4ef2af 100644
--- a/os_quirks.sh
+++ b/os_quirks.sh
@@ -18,6 +18,7 @@ case $OS in
         BUILD_LIBFFI="yes"
         BUILD_ZLIB="yes"
         BUILD_XZ="yes"
+        BUILD_LIBEDIT="yes"
         if [ -f /etc/alpine-release ]; then
             # The busybox ersatz binary on Alpine Linux is different.
             SHA_CMD=(sha512sum -csw)
@@ -50,6 +51,7 @@ case $OS in
         # System includes bzip2 libs by default.
         BUILD_BZIP2="no"
         BUILD_XZ="yes"
+        BUILD_LIBEDIT="yes"
         # Install package "p5-Digest-SHA" to get shasum binary.
         SHA_CMD=(shasum --algorithm 512 --check --status --warn)
         ;;
@@ -59,6 +61,7 @@ case $OS in
         # libffi not available in the base system, only as port/package.
         BUILD_LIBFFI="yes"
         BUILD_XZ="yes"
+        BUILD_LIBEDIT="yes"
         SHA_CMD=(sha512 -q -c)
         ;;
     sol*)
@@ -83,6 +86,7 @@ case $OS in
         # Multiple system libffi libs are present, this is a problem in 11.4.
         BUILD_LIBFFI="yes"
         BUILD_XZ="yes"
+        BUILD_LIBEDIT="yes"
         # Native tar is not that compatible, but the GNU tar should be present.
         TAR_CMD=(gtar xfz)
         ;;
diff --git a/pkg_checks.sh b/pkg_checks.sh
index a93cd60..cb35fbb 100644
--- a/pkg_checks.sh
+++ b/pkg_checks.sh
@@ -16,15 +16,17 @@

 # List of OS packages required for building Python/pyOpenSSL/cryptography etc.
 BASE_PKGS="gcc make m4 patch unzip perl"
-if [ "$BUILD_LIBEDIT" = "yes" ]; then
-    BASE_PKGS="$BASE_PKGS automake libtool"
-fi
 APK_PKGS="$BASE_PKGS git curl bash musl-dev linux-headers lddtree shadow \
     openssh-client file unzip g++ musl-locales dejagnu"
 DEB_PKGS="$BASE_PKGS tar diffutils git curl \
     openssh-client libtest-simple-perl xz-utils g++ dejagnu"
 RPM_PKGS="$BASE_PKGS tar diffutils git-core curl \
     openssh-clients perl-Test-Simple perl-IPC-Cmd xz gcc-c++ dejagnu"
+if [ "$BUILD_LIBEDIT" = "yes" ]; then
+    APK_PKGS="$APK_PKGS automake libtool ncurses-dev"
+    DEB_PKGS="$DEB_PKGS automake libtool libncurses5-dev"
+    RPM_PKGS="$RPM_PKGS automake libtool ncurses-devel"
+fi

 # Check for OS packages required for the build.
 MISSING_PACKAGES=""
diff --git a/src/libedit/chevahbs b/src/libedit/chevahbs
index b5d136e..fb3095d 100755
--- a/src/libedit/chevahbs
+++ b/src/libedit/chevahbs
@@ -18,16 +18,6 @@ chevahbs_getsources() {
 }

-chevahbs_patch() {
-    # With libedit, terminal state is not properly restored on some OS'es, eg.
-    # after exiting the repl on systems with ncurses 6.0, such as Ubuntu 16.04.
-    # Patch from https://trac.macports.org/ticket/48807#comment:141.
-    # Upstream bugfix (not recommended): https://gnats.netbsd.org/48957.
-    echo "# Applying terminal_restore.patch... #"
-    execute patch -p0 < terminal_restore.patch
-}
-
-
 chevahbs_configure() {
     # To properly quote multiple options, this is a Bash array.
     CONF_OPTS=(--disable-shared --enable-static)
diff --git a/src/libedit/sha512.sum b/src/libedit/sha512.sum
index fe121e7..d37ce1b 100644
--- a/src/libedit/sha512.sum
+++ b/src/libedit/sha512.sum
@@ -1 +1 @@
-597ed2282240be6dc3c4a7bb8b4ade31fe16ebfeb2229e3aec58643a65882653cc91deb300343d1b1cc588959dbf95698122c4005e47dfb947efa3e6b134bb82  libedit-20170329-3.1.tar.gz
+bc17371eeb8842b93cd5ed7ce3a04aa1cadf26aa697d92e3440f9f729a4d0631eef60ea2c96844ff773e1b3b80ae518fd3ae684126373dfc69b65d67a0f25e90  libedit-20240517-3.1.tar.gz

[adiroiban]

I think that python-prompt-toolkit is the best solution for us.

I don't know when I will have time for it, but I plan to use python-prompt-toolkit to implement history for chevah/server admin-shell