Can’t neither use nor compile static readline library

neodix42 commented 8 months ago

Describe the bug

I am trying to use pkgsStatic.readline in my project, however upon installation of one, the process fails with the message:

make[1]: Leaving directory '/build/readline/doc'
( cd examples ; make  DESTDIR=/nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy install )
make[1]: Entering directory '/build/readline/examples'
/bin/sh ../support/mkdirs /nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/share/readline
mkdir /nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/share/readline
/nix/store/ahkfdxq8mcpsb5kvdvgqr1wv8zjngbh4-coreutils-9.1/bin/install: cannot stat './rltest2.c': No such file or directory
make[1]: Leaving directory '/build/readline/examples'
/nix/store/ahkfdxq8mcpsb5kvdvgqr1wv8zjngbh4-coreutils-9.1/bin/install -c -m 644 /build/readline/readline.pc /nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/lib/pkgconfig/readline.pc
/nix/store/ahkfdxq8mcpsb5kvdvgqr1wv8zjngbh4-coreutils-9.1/bin/install -c -m 644 /build/readline/history.pc /nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/lib/pkgconfig/history.pc
mv /nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/lib/libreadline.a /nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/lib/libreadline.old
mv: cannot stat '/nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/lib/libreadline.a': No such file or directory
make: [Makefile:255: install-static] Error 1 (ignored)
/nix/store/ahkfdxq8mcpsb5kvdvgqr1wv8zjngbh4-coreutils-9.1/bin/install -c -m 644 libreadline.a /nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/lib/libreadline.a
test -n "ranlib" && ranlib /nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/lib/libreadline.a
mv /nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/lib/libhistory.a /nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/lib/libhistory.old
mv: cannot stat '/nix/store/zwkyjp5mz0vb6nlqjrbbsa1lkgvdwqd3-gnureadlinemy/usr/local/lib/libhistory.a': No such file or directory
make: [Makefile:258: install-static] Error 1 (ignored)

Steps To Reproduce

Actually, I tried to compile readline library directly from gnu git repository using the code below and got the same error (tried both nixos-unstable and 23.05, building on ubuntu 20.04)

export NIX_PATH=nixpkgs=https://github.com/nixOS/nixpkgs/archive/23.05.tar.gz


{ pkgs ? import <nixpkgs> { system = builtins.currentSystem; }
, stdenv ? pkgs.stdenv
, fetchgit ? pkgs.fetchgit
}:

stdenv.mkDerivation rec {
  name = "gnureadlinemy";

  src = fetchgit {
    url = "https://git.savannah.gnu.org/git/readline.git";
    rev = "refs/tags/readline-8.2";
    sha256 = "sha256-AT0WVrqbenjCFFo2mS/om3Y5jIRSJe4V8IEOyw324LM=";
  };

  nativeBuildInputs = with pkgs; [ automake libtool autoconf texinfo ];

  buildInputs = with pkgs; [ ];

  configurePhase = ''
   ./configure --enable-static --disable-shared
  '';

  installPhase = ''
    make install DESTDIR=$out
  '';
}

Expected behavior

Successful compilation and available static libreadline.

Screenshots

Additional context

Notify maintainers

Metadata

neodix42 commented 8 months ago

An interesting observation. Maybe it is not only static linking issue. If I change above the section and remove flags for ./configure the build fails with same error.

  configurePhase = ''
   ./configure
  '';

at the same time if I completely remove configurePhase section, the readline compiles ok, but no static library gets created.

lolbinarycat commented 8 months ago

instead of repackaging it, have you tried just modifying the src attribute of the existing libreadline derivation?

neodix42 commented 8 months ago

Thanks, I just tried your recommendation. Took that 8.2.nix file from your link (removed windows and meta parts) and it is the same - it compiles without specifying static option, but once I want to compile it statically it returns the same errors. Below is my version of the modified 8.2.nix file. Maybe I am setting that configurePhase wrongly...

{ pkgs ? import <nixpkgs> { system = builtins.currentSystem; }
, stdenv ? pkgs.stdenv
, lib ? pkgs.lib
, fetchgit ? pkgs.fetchgit
, fetchurl ? pkgs.fetchurl
, fetchpatch ? pkgs.fetchpatch
, ncurses ? pkgs.ncurses
, termcap ? pkgs.termcap
, curses-library ?
    if stdenv.hostPlatform.isWindows
    then termcap
    else ncurses
}:

stdenv.mkDerivation rec {
  pname = "readline";
  version = "8.2p${toString (builtins.length upstreamPatches)}";

  src = fetchgit {
    url = "https://git.savannah.gnu.org/git/readline.git";
    rev = "refs/tags/readline-8.2";
    sha256 = "sha256-AT0WVrqbenjCFFo2mS/om3Y5jIRSJe4V8IEOyw324LM=";
  };

#  src = fetchurl {
#    url = "mirror://gnu/readline/readline-${meta.branch}.tar.gz";
#    sha256 = "sha256-P+txcfFqhO6CyhijbXub4QmlLAT0kqBTMx19EJUAfDU=";
#  };

  configurePhase = ''
   ./configure --enable-static
  '';

  outputs = [ "out" "dev" "man" "doc" "info" ];
  strictDeps = true;
  propagatedBuildInputs = [ curses-library ];

  patchFlags = [ "-p0" ];

  upstreamPatches =
    (let
       patch = nr: sha256:
         fetchurl {
           url = "mirror://gnu/readline/readline-8.2-patches/readline82-${nr}";
           inherit sha256;
         };
     in
       import ./readline-8.2-patches.nix patch);

  patches = lib.optionals (curses-library.pname == "ncurses") [
    ./link-against-ncurses.patch
  ] ++ [
    ./no-arch_only-8.2.patch
  ]
  ++ upstreamPatches;
}

lolbinarycat commented 8 months ago

you shouldn't have to override configurePhase, try with configureFlags = [ "--enable-static" ];

wolfgangwalther commented 7 months ago

pkgsStatic.readline builds fine for me. What am I missing?

lolbinarycat commented 7 months ago

it also compiles for me now. @neodix42 can you confirm you still have this issue on the latest nixos stable channel?

neodix42 commented 7 months ago

I will double check in return to you within few days. Thank you guys!

neodix42 commented 7 months ago

You are right guys. I found a bug in our FindReadline.cmake file. However, still after adding pkgsStatic.readline.dev I receive an error:

/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0x45d9): undefined reference to 'tputs'

seems to be a common problem, but after adding pkgsStatic.ncurses.dev the same error appears. I have also tried to add linking flag -lncurses (didn't help) even though have paths to static libreadline.a. Btw, I am using makeStatic = true; in my nix file, maybe that causes a problem..

Is there any special handling of libreadline with combination of ncurses? Any advice is welcomed, otherwise you can close the topic. Thank you.

lolbinarycat commented 7 months ago

if the bug only happens with a certain combination of packages in buildInputs, it sounds like it's a problem with how the setupHook of those packages interact.

neodix42 commented 7 months ago

in my case buildInputs contain:

pkgsStatic.openssl microhttpdmy pkgsStatic.zlib pkgsStatic.libsodium.dev pkgsStatic.secp256k1 glibc.static
 pkgsStatic.readline.dev  pkgsStatic.ncurses.dev

I will take a look on how to adjust setupHook, thanks for the hint.

wolfgangwalther commented 7 months ago

pkgsStatic uses musl libc. You have glibc.static + a bunch of pkgsStatic dependencies listed. This looks like a recipe for failure to me.

neodix42 commented 7 months ago

Oh, that's a valuable information. I saw that musl libraries are used, but never connected that info to pkgsStatic usage, thought it is always by default, somehow. Thank you.

neodix42 commented 7 months ago

So I have tried to get rid of musl static libraries, but still got the same undefined reference to 'tputs', the final script looks like:

# export NIX_PATH=nixpkgs=https://github.com/nixOS/nixpkgs/archive/23.05.tar.gz

{ pkgs ? import <nixpkgs> { system = builtins.currentSystem; }
, lib ? pkgs.lib
, stdenv ? pkgs.stdenv
}:
let

  staticOptions = pkg: pkg.overrideAttrs(oldAttrs: {
    dontDisableStatic = true;
    enableSharedExecutables = false;
    configureFlags = (oldAttrs.configureFlags or []) ++ [ "--without-shared" "--disable-shared" "--disable-tests" ];
  });

  secp256k1my = (staticOptions pkgs.secp256k1);
  libsodiummy = (staticOptions pkgs.libsodium);
  readlinemy = (staticOptions pkgs.readline);
  ncursesmy = (staticOptions pkgs.ncurses);
  microhttpdmy = (staticOptions pkgs.libmicrohttpd);

in
stdenv.mkDerivation {
  pname = "prj";
  version = "dev-bin";

  src = ./.;

  nativeBuildInputs = with pkgs;
    [  cmake ninja git pkg-config  ];

  buildInputs = with pkgs;
    [
      (openssl.override { static = true; }).dev
      microhttpdmy
      (zlib.override { shared = false; }).dev
      secp256k1my
      libsodiummy
      readlinemy
      ncursesmy
      glibc.static
    ];

  makeStatic = true;

  LDFLAGS = [
     "-static-libgcc" "-static-libstdc++" "-static"
  ];
}

and the part of output:

/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(display.o): in function `update_line':
(.text+0x406b): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0x4135): undefined reference to `tgoto'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0x4149): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0x4305): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0x495f): undefined reference to `tgoto'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0x4971): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0x4b35): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0x4b97): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(display.o): in function `_rl_clear_screen':
(.text+0x70df): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(display.o): in function `_rl_clear_to_eol':
(.text+0x2176): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(display.o):(.text+0x7104): more undefined references to `tputs' follow
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(terminal.o): in function `_rl_get_screen_size':
(.text+0x163): undefined reference to `tgetnum'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0x1a2): undefined reference to `tgetnum'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(terminal.o): in function `_rl_backspace':
(.text+0x680): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(terminal.o): in function `rl_ding':
(.text+0x74a): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(terminal.o): in function `_rl_init_terminal_io':
(.text+0xb92): undefined reference to `PC'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0xbad): undefined reference to `BC'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0xbba): undefined reference to `UP'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0xc49): undefined reference to `tgetent'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0xc87): undefined reference to `tgetstr'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0xcb4): undefined reference to `PC'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0xccc): undefined reference to `BC'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0xce0): undefined reference to `UP'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0xcf5): undefined reference to `tgetflag'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0xd57): undefined reference to `tgetflag'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0xf29): undefined reference to `tgetflag'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(terminal.o): in function `_rl_enable_meta_key':
(.text+0x10c7): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(terminal.o): in function `_rl_disable_meta_key':
(.text+0x1129): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(terminal.o): in function `_rl_cr':
(.text+0x707): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(terminal.o): in function `_rl_standout_on':
(.text+0x7b3): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(terminal.o): in function `_rl_standout_off':
(.text+0x7f3): undefined reference to `tputs'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: /nix/store/8gfj3jnf74kqdl5djm7in8rygyjhg20x-readline-8.2p10/lib/libreadline.a(terminal.o):(.text+0x1049): more undefined references to `tputs' follow
collect2: error: ld returned 1 exit status

is this a bug or still a misuse?

wolfgangwalther commented 7 months ago

Instead of passing readlinemy to buildInputs, does readlinemy.dev work?

In any case - if I was trying to build a static executable, I'd probably try something like this:

# export NIX_PATH=nixpkgs=https://github.com/nixOS/nixpkgs/archive/23.05.tar.gz

{ pkgs ? import <nixpkgs> { system = builtins.currentSystem; } }:
let
  package =
  { cmake
  , git
  , libmicrohttpd
  , libsodium
  , ncurses
  , ninja
  , openssl
  , pkg-config
  , readline
  , secp256k1
  , stdenv
  , zlib
  }: stdenv.mkDerivation {
    pname = "prj";
    version = "dev-bin";

    src = ./.;

    nativeBuildInputs = [
      cmake
      ninja
      git
      pkg-config
    ];

    buildInputs = [
      libmicrohttpd
      libsodium
      ncurses
      openssl
      readline
      secp256k1
      zlib
    ];
  };
in
pkgs.pkgsStatic.callPackage package {}

This way, you could let pkgsStatic do the magic and not worry about all of that yourself.

neodix42 commented 6 months ago

Instead of passing readlinemy to buildInputs, does readlinemy.dev work?

still the same errors :

/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0x4135): undefined reference to `tgoto'
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: (.text+0x4149): undefined reference to `tputs'

with a very elegant solution from @wolfgangwalther where on contrary all packages are musl based also receiving the same error:

/nix/store/zq91mr9h32cippzbnn321vrmbqkh4mi5-x86_64-unknown-linux-musl-binutils-2.40/bin/x86_64-unknown-linux-musl-ld: (.text+0x414f): undefined reference to `tgoto'
/nix/store/zq91mr9h32cippzbnn321vrmbqkh4mi5-x86_64-unknown-linux-musl-binutils-2.40/bin/x86_64-unknown-linux-musl-ld: (.text+0x4163): undefined reference to `tputs'

tried here also to use readline.dev and/or ncurses.dev.

wolfgangwalther commented 6 months ago

where on contrary all packages are musl based also receiving the same error:

Ok, very good to confirm, that this wasn't about your static build setup, then.

Noting this comment:

# export NIX_PATH=nixpkgs=https://github.com/nixOS/nixpkgs/archive/23.05.tar.gz

Did you try with 23.11 or latest unstable, too?

neodix42 commented 6 months ago

I have noticed that within final binary linking stage the parameters -lcurses -lncurses or -ltermcap are not added, so I tried to add them manually, but then got an error:

/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: cannot find -lcurses: No such file or directory
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: cannot find -lncurses: No such file or directory
/nix/store/p0p56gzz837fgmfd7lyisghcd7x2fdlc-binutils-2.40/bin/ld: cannot find -ltermcap: No such file or directory

at the same time libcurses.a is present under /usr/lib/x86-64-linux-gnu/ and /nix/store/****-ncurses-static-x86-64-unknown-linux-musl-6.4/lib.

Anyway, I tried both with 23.11 and unstable releases with musl version and my one with overriden buildInputs and the results are below:

23.11 and unstable with overriden libs return same errors:

/nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: /nix/store/cgmvwvxjbrw2avqhakzkhz0mrq6cpkip-readline-8.2p1/lib/libreadline.a(display.o): in function update_line': (.text+0x4194): undefined reference totputs' /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: (.text+0x421f): undefined reference to tgoto' /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: (.text+0x4233): undefined reference totputs' /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: (.text+0x4666): undefined reference to `tputs'

23.11 and unstable with musl return also same errors:

/nix/store/w9dk8id4zmvmwqy42dyqc2k56nf14qvb-x86_64-unknown-linux-musl-binutils-2.40/bin/x86_64-unknown-linux-musl-ld: /nix/store/az4bx09l9db5xza97d66bkx50vyx07v6-readline-static-x86_64-unknown-linux-musl-8.2p1/lib/libreadline.a(display.o): in function `update_line':
(.text+0x3f85): undefined reference to `tgoto'
/nix/store/w9dk8id4zmvmwqy42dyqc2k56nf14qvb-x86_64-unknown-linux-musl-binutils-2.40/bin/x86_64-unknown-linux-musl-ld: (.text+0x3f99): undefined reference to `tputs'
/nix/store/w9dk8id4zmvmwqy42dyqc2k56nf14qvb-x86_64-unknown-linux-musl-binutils-2.40/bin/x86_64-unknown-linux-musl-ld: (.text+0x405b): undefined reference to `tputs'
/nix/store/w9dk8id4zmvmwqy42dyqc2k56nf14qvb-x86_64-unknown-linux-musl-binutils-2.40/bin/x86_64-unknown-linux-musl-ld: (.text+0x4518): undefined reference to `tputs'

This is the linking command of my execuatable named lite-client, I can see that all second and third parties libs are added, except libcurses.

FAILED: lite-client/lite-client
: && /nix/store/ac1hb5dc2z4biwgy8mjrhlifffkkrvdq-gcc-wrapper-13.2.0/bin/g++ -march=native -fstrong-eval-order=some -fno-omit-frame-pointer -ffunction-sections -fdata-sections -Wall -Wextra -Wimplicit-fallthrough=2 -Wpointer-arith -Wcast-qual -Wsign-compare -Wduplicated-branches -Wduplicated-cond -Walloc-zero -Wlogical-op -Wno-tautological-compare -Wpointer-arith -Wvla -Wnon-virtual-dtor -Wno-unused-parameter -Wconversion -Wno-sign-conversion -Wno-redundant-move -O3 -DNDEBUG -static-libgcc -static-libstdc++ -static -Wl,--gc-sections -Wl,--exclude-libs,ALL lite-client/CMakeFiles/lite-client.dir/lite-client.cpp.o -o lite-client/lite-client  tdutils/libtdutils.a  tdactor/libtdactor.a  adnl/libadnllite.a  tl/libtl_api.a  tl/libtl_lite_api.a  tl-utils/libtl-lite-utils.a  crypto/libton_crypto.a  crypto/libton_block.a  terminal/libterminal.a  lite-client/liblite-client-common.a  libgit.a  /nix/store/kkngxkszadrhxdzm6nf08jb9ziq9pjws-readline-8.2p10/lib/libreadline.a  adnl/libadnllite.a  tdnet/libtdnet.a  keys/libkeys.a  tl-utils/libtl-utils.a  tl-utils/libtl-lite-utils.a  tl/libtl_lite_api.a  crypto/libton_crypto.a  crypto/libton_block.a  crypto/libton_crypto.a  crypto/libton_block.a  crypto/libton_crypto_core.a  tddb/libtddb_utils.a  -lz  third-party/blst/libblst.a  /nix/store/6qwsqyh7jbqwijn6y8y006y1iwlcjmjy-secp256k1-0.4.1/lib/libsecp256k1.a  /nix/store/ha3vc38qr8lbss547pgj74gqyndg8dza-libsodium-1.0.19/lib/libsodium.a  tdactor/libtdactor.a  tl/libtl_api.a  tdutils/libtdutils.a  /nix/store/i42rzr64hz6qmqpkxgcm6msh4mzb4191-openssl-3.0.13/lib/libcrypto.a  -ldl  /nix/store/1dl21p2k8hjhvgirlykps3271p930cwx-zlib-1.3.1/lib/libz.a  third-party/crc32c/libcrc32c.a

wolfgangwalther commented 6 months ago

I suggest you try to use libedit instead of readline. It's a drop-in replacement, but with a significant difference for static linking:

The readline.pc file contains no Requires.private or Libs.private mention for -lncurses, but libedit.pc does have Libs.private: -lncurses in it. So this should work.

wolfgangwalther commented 6 months ago

The readline.pc file contains no Requires.private or Libs.private mention for -lncurses,

Ah, but it contains Requires.private: tinfo - and ncurses provides tinfo.pc, too, which should include -lncursesw. So this should actually work with readline, I guess...

neodix42 commented 6 months ago

As it usually happens, we can't replace readline with libedit that easy...

I did some analysis around readline.pc file.

I noticed that if readline is taken from the nix repository (musl or glibc) after compilation it does not have any readline.pc file under /nix/store/..../lib/pkgconfig directory. So I decided to compile gnu readline manually :


{ pkgs ? import <nixpkgs> { system = builtins.currentSystem; }
, stdenv ? pkgs.stdenv
, fetchgit ? pkgs.fetchgit
}:

stdenv.mkDerivation rec {
  name = "gnureadlinemy";

  src = fetchgit {
    url = "https://git.savannah.gnu.org/git/readline.git";
    rev = "refs/tags/readline-8.2";
    sha256 = "sha256-AT0WVrqbenjCFFo2mS/om3Y5jIRSJe4V8IEOyw324LM=";
  };

  nativeBuildInputs = with pkgs; [ automake libtool autoconf texinfo ];

  buildInputs = with pkgs; [ ];

  configurePhase = ''
   ./configure --enable-static --disable-shared --with-curses
  '';

  installPhase = ''
    runHook preInstall
    make install DESTDIR=$out
    runHook postInstall
  '';
}

and noticed that now it contains readline.pc (with ncurses flag):

cat /nix/store/a44ax6aqi6fs89xa3vbdrz60qnkpyqdl-gnureadlinemy/usr/local/lib/pkgconfig/readline.pc
prefix=/usr/local
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: Readline
Description: Gnu Readline library for command line editing
URL: http://tiswww.cwru.edu/php/chet/readline/rltop.html
Version: 8.2
Requires.private: ncurses
Libs: -L${libdir} -lreadline
Cflags: -I${includedir})

ok, and to be on safe side I have also did the same with ncurses (with --with-terms flag):

cat ncursesmy.nix
{ pkgs ? import <nixpkgs> { system = builtins.currentSystem; }
, stdenv ? pkgs.stdenv
, fetchgit ? pkgs.fetchgit
}:

stdenv.mkDerivation rec {
  name = "ncursesmy";

  src = fetchgit {
    url = "https://github.com/mirror/ncurses.git";
    rev = "refs/tags/v6.4";
    sha256 = "sha256-Go3YvVaRzaL2t0AQ8UaEXxlW0U5J6c8eqprWP4z3S78=";
  };

  nativeBuildInputs = with pkgs; [ automake libtool autoconf texinfo ];

  buildInputs = with pkgs; [ ];

  configurePhase = ''
   ./configure --enable-static --disable-shared --with-terms
  '';

  installPhase = ''
    runHook preInstall
    make install DESTDIR=$out
    runHook postInstall
  '';
}

and specified all the paths via cmake parameters (final file):


{ pkgs ? import <nixpkgs> { system = builtins.currentSystem; }
, lib ? pkgs.lib
, stdenv ? pkgs.stdenv
}:
let
  gnureadlinemy = (import ./gnureadlinemy.nix) {};
  ncursesmy = (import ./ncursesmy.nix) {};
  staticOptions = pkg: pkg.overrideAttrs(oldAttrs: {
    dontDisableStatic = true;
    enableSharedExecutables = false;
    configureFlags = (oldAttrs.configureFlags or []) ++ [ "--without-shared" "--disable-shared" "--disable-tests" ];
  });

  secp256k1my = (staticOptions pkgs.secp256k1);
  libsodiummy = (staticOptions pkgs.libsodium);

  microhttpdmy = pkgs.libmicrohttpd.overrideAttrs(oldAttrs: {
    dontDisableStatic = true;
    enableSharedExecutables = false;
    configureFlags = (oldAttrs.configureFlags or []) ++ [ "--enable-static" "--disable-tests" "--disable-benchmark" "--disable-shared" "--disable-https" "--with-pic" ];
  });

in
stdenv.mkDerivation {
  pname = "ton";
  version = "dev-bin";

  src = ./.;

  nativeBuildInputs = with pkgs;
    [  cmake ninja git pkg-config  ];

  buildInputs = with pkgs;
    [
      (openssl.override { static = true; }).dev
      microhttpdmy.dev
      (zlib.override { shared = false; }).dev
      secp256k1my
      libsodiummy.dev
      gnureadlinemy
      ncursesmy
      glibc.static
    ];

  cmakeFlags = [
    "-DTON_USE_ABSEIL=OFF"
    "-DBUILD_SHARED_LIBS=OFF"
    "-DREADLINE_FOUND=1"
    "-DREADLINE_INCLUDE_DIR=${gnureadlinemy}/usr/local/include"
    "-DREADLINE_LIBRARY=${gnureadlinemy}/usr/local/lib/libreadline.a"
    "-DNCURSES_FOUND=1"
    "-DNCURSES_INCLUDE_DIR=${ncursesmy}/usr/include"
    "-DNCURSES_LIBRARY=${ncursesmy}/usr/lib/libncurses.a"
  ];

  makeStatic = true;
  doCheck = false;

  LDFLAGS = [
     "-static-libgcc" "-static-libstdc++" "-static"
  ];
}

and the final linking command is "right" (containing ncurses and readline):

/nix/store/7wkshj58fcsl1f3zyi67qsxgl1p8nki1-gcc-wrapper-12.2.0/bin/g++ -march=native -fstrong-eval-order=some -fno-omit-frame-pointer -ffunction-sections -fdata-sections -Wall -Wextra -Wimplicit-fallthrough=2 -Wpointer-arith -Wcast-qual -Wsign-compare -Wduplicated-branches -Wduplicated-cond -Walloc-zero -Wlogical-op -Wno-tautological-compare -Wpointer-arith -Wvla -Wnon-virtual-dtor -Wno-unused-parameter -Wconversion -Wno-sign-conversion -Wno-redundant-move -O3 -DNDEBUG -static-libgcc -static-libstdc++ -static -Wl,--gc-sections -Wl,--exclude-libs,ALL lite-client/CMakeFiles/lite-client.dir/lite-client.cpp.o -o lite-client/lite-client tdutils/libtdutils.a tdactor/libtdactor.a adnl/libadnllite.a tl/libtl_api.a tl/libtl_lite_api.a tl-utils/libtl-lite-utils.a crypto/libton_crypto.a crypto/libton_block.a terminal/libterminal.a lite-client/liblite-client-common.a libgit.a /nix/store/hqf4bd14zfxzrmqv5kmny3rwc2zzq9nm-ncursesmy/usr/lib/libncurses.a /nix/store/7x7ppba6sdv5nw650d0jqlpvbcj8nm79-gnureadlinemy/usr/local/lib/libreadline.a adnl/libadnllite.a tdnet/libtdnet.a keys/libkeys.a tl-utils/libtl-utils.a tl-utils/libtl-lite-utils.a tl/libtl_lite_api.a crypto/libton_crypto.a crypto/libton_block.a crypto/libton_crypto.a crypto/libton_block.a crypto/libton_crypto_core.a tddb/libtddb_utils.a -lz third-party/blst/libblst.a /nix/store/lq3bdn2c0q0radxz1pdqyhp5iqhjvzn6-secp256k1-0.3.2/lib/libsecp256k1.a /nix/store/anibb8mn9bwldxkdspcnjyjyy5l5y53v-libsodium-1.0.18/lib/libsodium.a tdactor/libtdactor.a tl/libtl_api.a tdutils/libtdutils.a /nix/store/7c9ib3kjd75cbyr1ql8xdjl9yp0j4vnw-openssl-3.0.8/lib/libcrypto.a -ldl /nix/store/jpqh4saqljf5b6yb6jf7xhizp9fm791c-zlib-1.2.13/lib/libz.a third-party/crc32c/libcrc32c.a

but at the end still same missing reference to tputs..

Obviously I am missing the understanding how to let readline know about ncurses :/

/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /nix/store/7x7ppba6sdv5nw650d0jqlpvbcj8nm79-gnureadlinemy/usr/local/lib/libreadline.a(complete.o): in function `rl_username_completion_function':
/build/readline/complete.c:2291: warning: Using 'getpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /nix/store/7x7ppba6sdv5nw650d0jqlpvbcj8nm79-gnureadlinemy/usr/local/lib/libreadline.a(tilde.o): in function `tilde_expand_word':
/build/readline/./tilde.c:386: warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /nix/store/7x7ppba6sdv5nw650d0jqlpvbcj8nm79-gnureadlinemy/usr/local/lib/libreadline.a(shell.o): in function `sh_get_home_dir':
/build/readline/shell.c:168: warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /nix/store/7x7ppba6sdv5nw650d0jqlpvbcj8nm79-gnureadlinemy/usr/local/lib/libreadline.a(complete.o): in function `rl_username_completion_function':
/build/readline/complete.c:2286: warning: Using 'setpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /build/readline/complete.c:2302: warning: Using 'endpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: tdutils/libtdutils.a(IPAddress.cpp.o): in function `td::IPAddress::init_host_port(td::CSlice, td::CSlice, bool)':
IPAddress.cpp:(.text._ZN2td9IPAddress14init_host_portENS_6CSliceES1_b+0x30b): warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /nix/store/7c9ib3kjd75cbyr1ql8xdjl9yp0j4vnw-openssl-3.0.8/lib/libcrypto.a(libcrypto-lib-bio_sock.o): in function `BIO_gethostbyname':
(.text+0x71): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /nix/store/7x7ppba6sdv5nw650d0jqlpvbcj8nm79-gnureadlinemy/usr/local/lib/libreadline.a(display.o): in function `_rl_move_cursor_relative':
/build/readline/display.c:2842: undefined reference to `tputs'
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /nix/store/7x7ppba6sdv5nw650d0jqlpvbcj8nm79-gnureadlinemy/usr/local/lib/libreadline.a(display.o): in function `_rl_move_vert':
/build/readline/display.c:2894: undefined reference to `tputs'
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /nix/store/7x7ppba6sdv5nw650d0jqlpvbcj8nm79-gnureadlinemy/usr/local/lib/libreadline.a(display.o): in function `delete_chars':
/build/readline/display.c:3301: undefined reference to `tputs'
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /nix/store/7x7ppba6sdv5nw650d0jqlpvbcj8nm79-gnureadlinemy/usr/local/lib/libreadline.a(display.o): in function `open_some_spaces':
/build/readline/display.c:3258: undefined reference to `tgoto'
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /build/readline/display.c:3259: undefined reference to `tputs'
/nix/store/22p5nv7fbxhm06mfkwwnibv1nsz06x4b-binutils-2.40/bin/ld: /build/readline/display.c:3278: undefined reference to `tputs'

wolfgangwalther commented 6 months ago

I noticed that if readline is taken from the nix repository (musl or glibc) after compilation it does not have any readline.pc

You need to look into the readline.dev output. The readline.pc file is there, but it doesn't have ncurses in Requires.private. Not sure why, yet.

One thing I wonder: Are we actually talking about a "static only" problem here? I.e. does my simplified build setup in https://github.com/NixOS/nixpkgs/issues/288624#issuecomment-2048036106 work when you replace the final pkgs.pkgsStatic.callPackage call with pkgs.callPackage? Does that build a dynamically linked executable successfully?

neodix42 commented 6 months ago

Actually, I always build a shared version natively with clang + ninja. Now tried your simplified nix script with pkgs.callPackage and the whole project was built successfully.

NixOS / nixpkgs