Open dontlaugh opened 6 months ago
dontlaugh
commented
6 months ago A workaround is to compile without readline and use rlwrap
./configure --without-readline
./_build/oils.sh
doas ./installNeeds hacks from #1943 to build/install. After that, invoke with rlwrap
% rlwrap osh
osh-0.21.0$ echo $HOME
/home/coleman
osh-0.21.0$ 
andychu
commented
6 months ago Hm do you know what version of GNU readline is on OpenBSD?
I am not sure what versions we support, but we should document it
It looks like the current version is 8.2 - https://tiswww.case.edu/php/chet/readline/rltop.html
dontlaugh
commented
6 months ago Looks like 8.2 on my system:
% pkg_info readline
Information for https://ftp.openbsd.org/pub/OpenBSD/7.5/packages/aarch64/readline-8.2.tgz
Comment:
library to edit command lines as they are typed in
Description:
The GNU Readline library provides a set of functions for use by
applications that allow users to edit command lines as they are typed
in. Both Emacs and vi editing modes are available. The Readline library
includes additional functions to maintain a list of previously-entered
command lines to recall and perhaps reedit those lines, and perform
csh-like history expansion on previous commands.
The history facilities are also placed into a separate library, the
History library, as part of the build process. The History library may
be used without Readline in applications which desire its capabilities.
To avoid any confusion with the base readline/history library make sure
you link to ereadline and ehistory. In the headers you can check for
RL_READLINE_VERSION, RL_VERSION_MAJOR and RL_VERSION_MINOR to see which
header you have picked up.
Maintainer: The OpenBSD ports mailing-list <ports@openbsd.org>
WWW: https://tiswww.case.edu/php/chet/readline/rltop.htmlHmm I'm surprised that a newer version is not working... I have readline 7.0 on my Ubuntu machine
I wonder if this is reproducible on Linux with readline 8.2
andychu
commented
6 months ago Oh I wonder if you can do
$ head _build/detected*
==> _build/detected-config.h <==
#define SIZEOF_INT 4
#define SIZEOF_LONG 8
#define SIZEOF_VOID_P 8
#define SIZEOF_SHORT 2
#define SIZEOF_FLOAT 4
#define SIZEOF_DOUBLE 8
#define SIZEOF_SIZE_T 8
#define SIZEOF_FPOS_T 16
#define SIZEOF_PID_T 4
#define SIZEOF_OFF_T 8
==> _build/detected-config.sh <==
HAVE_READLINE=1
READLINE_DIR=
PREFIX=/usr/local
DATAROOTDIR=/usr/local/share
STRIP_FLAGS=--gc-sections
==> _build/detected-cpp-config.h <==
#define HAVE_READLINE 1
#define HAVE_PWENT 1That will show what your HAVE_READLINE setting is
Maybe that test is broken on OpenBSD somehow?
dontlaugh
commented
6 months ago ~I think you might be onto something~ Doh! I forgot I had files from --without-readline on disk. Here is a new configure run.
0 % ./configure
./configure: Wrote _build/detected-cpp-config.h
./configure: Wrote _build/detected-config.sh
./configure: Wrote _build/detected-config.h
0 % head _build/detected*
==> _build/detected-config.h <==
#define SIZEOF_INT 4
#define SIZEOF_LONG 8
#define SIZEOF_VOID_P 8
#define SIZEOF_SHORT 2
#define SIZEOF_FLOAT 4
#define SIZEOF_DOUBLE 8
#define SIZEOF_SIZE_T 8
#define SIZEOF_FPOS_T 8
#define SIZEOF_PID_T 4
#define SIZEOF_OFF_T 8
==> _build/detected-config.sh <==
HAVE_READLINE=1
READLINE_DIR=
PREFIX=/usr/local
DATAROOTDIR=/usr/local/share
STRIP_FLAGS=--gc-sections
==> _build/detected-cpp-config.h <==
#define HAVE_READLINE 1
#define HAVE_PWENT 1There still might be something wrong with feature detection, but I'm not sure what it is. OpenBSD puts the headers here:
% ls /usr/include/readline
chardefs.h history.h keymaps.h readline.h rlconf.h rlstdc.h rltypedefs.h tilde.hI wonder if this is reproducible on Linux with readline 8.2
Not on my Linux laptop. I have compiled Oils without any issues on my Void Linux machine, and it has readline 8.2
Perhaps OpenBSD patches readline.
dontlaugh
commented
6 months ago okay whoa I asked on libera #openbsd and got the lowdown.
pkg_info is not showing my current readline. the readline in BASE is much older
#define RL_READLINE_VERSION 0x0403 /* Readline 4.3 */So I found some more info. The readline in base is quite old, and a lot of gnu stuff in OpenBSD base is old.
However, readline 8.2 is available from the ports tree.
https://github.com/openbsd/ports/blob/master/devel/readline/Makefile#L31
I installed it with
doas pkg_add readlineAnything not in base gets installed under /usr/local. As you can see from the ports tree's makefile for readline, they also rename the library to ereadline. So some build script updates will need to happen.
Headers get installed here:
ls /usr/local/include/ereadline/readline
chardefs.h history.h keymaps.h readline.h rlconf.h rlstdc.h rltypedefs.h tilde.hExample programs (?) here
ls /usr/local/share/readline
excallback.c fileman.c histexamp.c manexamp.c rl-fgets.c rl.c rlbasic.c rlcat.c rlevent.c rlptytest.c rltest.c rlversion.cAnd libraries here
ls /usr/local/lib | grep readline
libereadline.a
libereadline.so.3.0And it's got the symbols
So it seems like you need to include/link against ereadline on OpenBSD.
dontlaugh
commented
6 months ago Here is the ports entry https://github.com/openbsd/ports/blob/9c275c5eec69b00ca423afc88f72f39a640dfb1d/devel/readline/pkg/DESCR#L12
ereadline and perhaps also ehistory will be required.
On the other hand, pkg-config shows this.
% pkg-config --libs readline
-L/usr/local/lib -lreadline
% pkg-config --libs history
-L/usr/local/lib -lhistory
% pkg-config --libs ereadline
Package ereadline was not found in the pkg-config search pathUpdate: currently manually hacking the readline paths in build/ninja-rules-cpp.sh to see if I can get oils to compile. That can help determine what changes the configure script needs, if any.
Update 2: adding -lreadline to $link_flags works, but -lhistory fails to link. -I/usr/local/include/ereadline seems to find the headers.
andychu
commented
6 months ago Oh thanks for trying it! Yes I'd be interested in the diff of what works
I don't have an OpenBSD machine to test on
Do you need -lhistory? is that separate on OpenBSD?
dontlaugh
commented
6 months ago Do you need -lhistory? is that separate on OpenBSD?
Yes, according to the DESC file in the ports entry for readline:
The history facilities are also placed into a separate library, the History library, as part of the build process. The History library may be used without Readline in applications which desire its capabilities.
To avoid any confusion with the base readline/history library make sure you link to ereadline and ehistory. In the headers you can check for RL_READLINE_VERSION, RL_VERSION_MAJOR and RL_VERSION_MINOR to see which header you have picked up.
I'm having some trouble getting it to link, though. Even though the compiled libraries are right next to the readline ones
% ls /usr/local/lib | grep history
libehistory.a
libehistory.so.3.0Oh it has to be -lehistory not -lhistory ? confusingly the flag has to match whatever's after lib I think
dontlaugh
commented
6 months ago Not sure :( pkg-config seems to indicate it's -lhistory
I might have to try a simplified C program to see precisely what's required to include these libraries.
andychu
commented
5 months ago Yeah I agree the best solution here is to produce a self-contained C program on OpenBSD that links GNU readline and calls a few functions
We have a similar problem on FreeBSD
I would like this to work, but I don't have access to either OpenBSD or FreeBSD machines
Description of how Oils works here:
https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/installing.20on.20FreeBSD
So if there's a simple C program example, then it can probably ported over to our build system pretty easily
andychu
commented
5 months ago Or I guess another solution is to use pkg-config ? Is that the standard on OpenBSD ?
I'm unclear about this
dontlaugh
commented
5 months ago I'm not sure if pkg-config is standard, but some libraries drop .pc files and stuff.
Regarding CI it seems like SourceHut has FreeBSD and OpenBSD images. I've been shopping around for a solution, myself, since I've become BSD-curious.
I haven't had time to work this issue in particular, but I haven't forgotten. Feel free to poke me again if I don't get back to you in ... a few weeks? :heart:
greggyb
commented
5 months ago I'm planning on getting a POC C program with readline on FreeBSD (from the Zulip thread Andy linked earlier), probably this weekend, in a Sourcehut build. I'll also give OpenBSD a try at the same time. And I'll update this issue with my results.
POC program and compilation: https://oilshell.zulipchat.com/#narrow/stream/121540-oil-discuss/topic/installing.20on.20FreeBSD/near/437738525
POC program and compilation copied below:
#include <stdio.h>
#include <stdlib.h>
#include "readline/readline.h"
int main() {
char *line;
line = readline("Enter your name: ");
printf("Hello, %s!\n", line);
free(line);
return 0;
}# FreeBSD compilation
$ cc -I/usr/local/include -L/usr/local/lib -l readline r.c -o r
$ ./r
Enter your name: a
Hello, a!Thanks for the test program. Here's what happened when I built it on OpenBSD with the system readline and the one from ports.
-l readline links against the system readline.
% cc -I/usr/local/include -L/usr/local/lib -l readline r.c -o r
% ldd r
r:
Start End Type Open Ref GrpRef Name
0000000829f40000 0000000829f71000 exe 1 0 0 r
0000000d29e3b000 0000000d29eab000 rlib 0 1 0 /usr/lib/libreadline.so.5.0
0000000d043b2000 0000000d044dd000 rlib 0 1 0 /usr/lib/libc.so.99.0
0000000c6a7bd000 0000000c6a864000 rlib 0 1 0 /usr/lib/libcurses.so.15.0
0000000d17be0000 0000000d17be0000 ld.so 0 1 0 /usr/libexec/ld.so-l ereadline links against the readline from ports.
% cc -I/usr/local/include -L/usr/local/lib -l ereadline r.c -o r
% ldd r
r:
Start End Type Open Ref GrpRef Name
0000001e0f690000 0000001e0f6c1000 exe 1 0 0 r
00000022998df000 000000229996e000 rlib 0 1 0 /usr/local/lib/libereadline.so.3.0
000000227a3b6000 000000227a4e1000 rlib 0 1 0 /usr/lib/libc.so.99.0
0000002304d56000 0000002304dfd000 rlib 0 1 0 /usr/lib/libcurses.so.15.0
00000022f81e0000 00000022f81e0000 ld.so 0 1 0 /usr/libexec/ld.soBoth versions executed fine.
andychu
commented
4 months ago OK thanks, this is helpful !
So I guess the first solution is for our build scripts to provide arbitrary control over -I -L -l or something
Right now we have this:
~/git/oilshell/oil$ ./configure --help
Usage: ./configure [OPTION...]
Detects system settings before a build of Oil.
Installation directories:
--prefix=PREFIX Prefix for the bin/ directory [/usr/local]
--datarootdir=DATAROOTDIR Prefix for data files, including man page [PREFIX/share]
Optional features:
--with-readline Fail unless readline is available.
--without-readline Don't compile with readline, even if it's available.
The shell won't have any interactive features.
--readline=DIR An alternative readline installation to link against
--with-systemtap-sdt Fail unless systemtap-sdt is available.
--without-systemtap-sdt Don't compile with systemtap-sdt, even if it's available.which I guess is not enough ??
We are trying to follow GNU configure standards , but with a hand-written shell script, not autoconf
https://www.gnu.org/prep/standards/html_node/Configuration.html
Question -- does anyone have any shell snippets that show how bash is configured on FreeBSD and OpenBSD?
What flags are passed?
I think OSH should look the same way
(Unless bash is special somehow, because GNU readline and bash are maintained together ...)
dontlaugh
commented
4 months ago The bash shell is in ports, not base. Here it is in the ports tree: https://github.com/openbsd/ports/tree/master/shells/bash
None of the flags mention readline or ereadline :thinking:
Here's some inpection of the bash executable with nm, ldd, and grep.
The bash we're dealing with.
% which bash
/usr/local/bin/bashSearching specifically for the missing symbols in my original compiler error
% nm $(which bash) | grep free_history
0014d2a0 T _rl_free_history_entry
0014e860 T free_history_entry% nm $(which bash) | grep clear_history
000e5a64 T bash_clear_history
0014f220 T clear_history
0014d780 T rl_clear_history% nm $(which bash) | grep sigcleanup
00131fb8 t _rl_complete_sigcleanup
0018e8a8 B _rl_sigcleanup
00147650 T rl_callback_sigcleanupThe output T indicates an external symbol, but I actually don't know if that guarantees that the readline implementation is not embedded. I am going to go through fetching the sources from ports.
Some more wide-ranging searches for good measure.
So that's bash. The libraries we want (I think) are both from ports and are located in /usr/local/lib/
% nm /usr/local/lib/libehistory.so.3.0 | grep free_history_entry
00014144 T free_history_entry
% nm /usr/local/lib/libereadline.so.3.0 | grep free_history_entry
0005ab70 T _rl_free_history_entry
0005c1a4 T free_history_entry
% nm /usr/local/lib/libereadline.so.3.0 | grep rl_callback_sigcleanup
00054e38 T rl_callback_sigcleanup
% nm /usr/local/lib/libereadline.so.3.0 | grep rl_clear_history
0005b058 T rl_clear_historypkg-config does point towards the history and readline implementations we want.
% pkg-config --libs readline
-L/usr/local/lib -lreadline
% pkg-config --libs history
-L/usr/local/lib -lhistoryNote that ehistory and ereadline can't be found by pkg-config
% pkg-config --libs ehistory
Package ehistory was not found in the pkg-config search path
% pkg-config --libs ereadline
Package ereadline was not found in the pkg-config search pathHowever, as the compilation of the test program demonstrates, providing -L/usr/local/lib -lreadline will prefer the system readline, probably because it finds it first.
dontlaugh
commented
4 months ago Here is a new test program that includes some functions from history.h
#include <stdio.h>
#include <stdlib.h>
#include <readline/readline.h>
#include <readline/history.h>
void cb(char * line) {/* no-op */};
int main() {
char *input;
rl_initialize();
// from readline.h
rl_clear_history();
// from history.h
clear_history();
while ((input = readline(">>> ")) != NULL) {
if (*input) {
add_history(input);
}
if (strlen(input) == 3) {
HIST_ENTRY *e = remove_history(0);
free_history_entry(e);
}
printf("You entered: %s\n", input);
free(input);
}
rl_callback_sigcleanup();
return 0;
}This compiles with no warnings on OpenBSD (with ports readline installed)
% cc -I/usr/local/include/ereadline -L/usr/local/lib -l ereadline -l ehistory rltest.c -o rltestNote the -I flag. This compiles with warnings.
% cc -I/usr/local/include -L/usr/local/lib -l ereadline -l ehistory rltest.c -o rltest
rltest.c:13:5: warning: call to undeclared function 'rl_clear_history'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
rl_clear_history();
^
rltest.c:22:13: warning: call to undeclared function 'free_history_entry'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
free_history_entry(e);
^
rltest.c:27:5: warning: call to undeclared function 'rl_callback_sigcleanup'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
rl_callback_sigcleanup();
^
3 warnings generated.And this test program requires -l ereadline and -ehistory. -lreadline and -lhistory don't work afaict. -lhistory is not found, and -lreadline is missing symbols.
% cc -I/usr/local/include/ereadline -L/usr/local/lib -l readline -l history rltest.c -o rltest
ld: error: unable to find library -lhistory
cc: error: linker command failed with exit code 1 (use -v to see invocation)
% cc -I/usr/local/include/ereadline -L/usr/local/lib -l readline -l ehistory rltest.c -o rltest
ld: error: undefined symbol: rl_clear_history
>>> referenced by rltest.c
>>> /tmp/rltest-7307dc.o:(main)
ld: error: undefined symbol: rl_callback_sigcleanup
>>> referenced by rltest.c
>>> /tmp/rltest-7307dc.o:(main)
cc: error: linker command failed with exit code 1 (use -v to see invocation)
So, in summary, readline on OpenBSD needs
-I/usr/local/include/ereadline -L/usr/local/lib-lereadline-lehistorydoas pkg_add readlineI'll try to play with the configure script a bit so see if I can make it work on 0.21.0; 0.22.0 needs #1991
Update: Success?!
I got it to compile with these hacks applied directly to the 0.21.0 source tarball. Note that I still needed to apply the fix from #1943 too
diff --git a/_build/oils.sh b/_build/oils.sh
index 7b66bdf..54a311c 100755
--- a/_build/oils.sh
+++ b/_build/oils.sh
@@ -220,7 +220,7 @@ main() {
cd $out_dir
for symlink in osh ysh; do
- ln -s -f -v $out_name $symlink
+ ln -s -f $out_name $symlink
done
}
diff --git a/build/ninja-rules-cpp.sh b/build/ninja-rules-cpp.sh
index 029a27a..d427090 100755
--- a/build/ninja-rules-cpp.sh
+++ b/build/ninja-rules-cpp.sh
@@ -226,7 +226,7 @@ setglobal_link_flags() {
*)
if test "$HAVE_READLINE" = 1; then
- link_flags="$link_flags -lreadline"
+ link_flags="$link_flags -L/usr/local/lib -lereadline -lehistory"
fi
if test -n "$READLINE_DIR"; then
link_flags="$link_flags -L${READLINE_DIR}/lib"
@@ -357,7 +357,7 @@ symlink() {
local out=$3
cd $dir
- ln -s -f -v $in $out
+ ln -s -f $in $out
}
# test/cpp-unit.sh sources thisI rely on CXXFLAGS for -I since it gets passed through
export CXXFLAGS='-I/usr/local/include/ereadline
./configure
_build/oils.shAnd a quick test of osh shows that readline history is working.
% _bin/cxx-opt-sh/oils-for-unix osh
osh-0.21.0$ echo do i have history
do i have history
osh-0.21.0$ echo yes i do
yes i do
^Ch-0.21.0$ echo yes i do
osh-0.21.0$ exit
System Details
Error
The compiler starts erroring due to some missing symbols related to readline.
~The
rl_clear_historyfunction seems to be calledclear_historyin readline/readline.h.~ Oops that's Oils codeRipgreppin' through /usr led me to this comment in OpenBSD's Bash NEWS file (a changelog)
So there seem to be a lot of differences related to signal handling, so they ought to be approached with care.