Support for plpython?

[fitcs01]

ysqlsh (11.2-YB-2.5.3.1-b0) Type "help" for help.

yugabyte=# create extension plpython3u; ERROR: CREATE LANGUAGE not supported yet HINT: Please report the issue on https://github.com/YugaByte/yugabyte-db/issues

Any Plans to support language extensions?

Cheers, John

[FranckPachot]

See https://github.com/yugabyte/yugabyte-db/issues/12119

[crashbandicode]

After a lot of debugging I got this working by building yugabytedb with the details mentioned in https://github.com/yugabyte/yugabyte-db/commit/c809ee076f6cbcb1db3f811c28d1dcb6d7c5e42b

I was trying to run it from Docker so what I did was the following using an almalinux:8 container.

dnf -y update
dnf groupinstall -y 'Development Tools'
dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
dnf -y install epel-release libatomic rsync python3-devel cmake3 java-1.8.0-openjdk maven npm golang gcc-toolset-12 gcc-toolset-12-libatomic-devel patchelf glibc-langpack-en ccache vim wget python3.11-devel python3.11-pip clang ncurses-devel readline-devel libsqlite3x-devel
mkdir /opt/yb-build
chown "$USER" /opt/yb-build
alternatives --remove-all python3
alternatives --remove-all python
alternatives --install /usr/bin/python python /usr/bin/python3.11 3
alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 3
latest_zip_url=$(curl -Ls "https://api.github.com/repos/ninja-build/ninja/releases/latest" \
                 | grep browser_download_url | grep ninja-linux.zip | cut -d \" -f 4)
curl -Ls "$latest_zip_url" | zcat | tee /usr/local/bin/ninja >/dev/null
chmod +x /usr/local/bin/ninja
export YB_CCACHE_DIR="$HOME/.cache/yb_ccache"
# add #include "pg_yb_utils.h" to src/postgres/src/pl/plpython/plpy_procedure.c
# if using python > 3.9 remove #include <compile.h> and  #include <eval.h> from src/postgres/src/pl/plpython/plpython.h
# add '--with-python', to python/yugabyte/build_postgres.py under the configure_postgres method

./yb_build.sh -j$(nproc) --clean-all --build-yugabyted-ui --no-linuxbrew --clang15 -f release
...
CREATE OR REPLACE LANGUAGE plpython3u;

A few of the packages were for building python (which was something I tried) so you may or may not need them. Hopefully this helps someone in the future. There may be a way to do this without having to build yugabytedb but I couldn't figure it out. There also appears to be a fix for plpython3u when you use the bundled version per the link above.

[FranckPachot]

Thanks a lot @crashbandicode, I just tested and it works (~ 42 minutes to build, used 15GB on disk)

For full automation I've scripted the changes to the files:

# add #include "pg_yb_utils.h" to src/postgres/src/pl/plpython/plpy_procedure.c
sed -e '/#include "postgres.h"/a#include "pg_yb_utils.h"' -i src/postgres/src/pl/plpython/plpy_procedure.c

# if using python > 3.9 remove #include <compile.h> and  #include <eval.h> from src/postgres/src/pl/plpython/plpython.h
sed -e '/#include <compile.h>/d' -e '/#include <eval.h>/d' -i src/postgres/src/pl/plpython/plpython.h

# add '--with-python', to python/yugabyte/build_postgres.py under the configure_postgres method
sed -e "/'\.\/configure',/a\                '--with-python'," -i python/yugabyte/build_postgres.py

This is now easy to automate in a Dockerfile

There may be a way to do this without having to build yugabytedb but I couldn't figure it out.

I'll check with engineering.

[FranckPachot]

Note that I have an error with yb_release:

#0 2187.3 RuntimeError: Multiple dependencies with the same name libc.so.6 but different targets: [Dependency(name='libc.so.6', target='/opt/yb-build/brew/linuxbrew-20181203T161736v9-3ba4c2ed9b0587040949a4a9a95b576f520bae/Cellar/glibc/2.23/lib/libc-2.23.so', origin='/yugabyte-db/build/release-clang15-linuxbrew-dynamic-ninja/bin/ldb'), Dependency(name='libc.so.6', target='/usr/lib64/libc-2.28.so', origin='/yugabyte-db/build/release-clang15-linuxbrew-dynamic-ninja/postgres/lib/plpython3.so')]

[crashbandicode]

Yeah I saw this issue too. Forgot about it because I ran yugabyte directly after building instead of running the release script. To fix this you need to cd into the bin directory, do a chmod +x * and then run a YB_USE_LINUXBREW=0 ./yb_release and it should build. There's only three executables without execute permission (get_clients.sh, parse_contention.py, and yb-check-consistency.py) so you can do it explicitly as well if you want. I just tested it and it completed successfully on my end without needed to clear anything out. For context on disabling linuxbrew I found this bug which gave me the idea: https://github.com/yugabyte/yugabyte-db/issues/5529 .

[FranckPachot]

@crashbandicode can you ping me at fpachot@yugabyte.com (or twitter, linkedin...) please, I would like to thank you for the time you made me save 😀