search

pololu / nixcrpkgs

Tools for cross-compiling standalone applications using Nix.
Other
200 stars 17 forks source link

Adding libxslt as a cross_input appears to not work. #11

Open seanybaggins opened 1 year ago

seanybaggins commented 1 year ago

Here is my nix/build.nix

src: config_name: env:
let
  payload = env.make_derivation {
    builder = ../Scripts/builder.sh;
    inherit src;
    native_inputs = with env.nixpkgs; [
      git
      bear
      coreutils
      env.qt
      curl.dev
      libxslt.dev
    ];
    cross_inputs = with env.nixpkgs; [
      env.qt
      libxslt.dev
      sol2
      lua
      libusb1
      sqlcipher
      xmlsec
      avahi
      boost171
      libtool
    ];
  };

  license_set = env.libusbp.license_set // env.global_license_set;

  license = env.make_derivation {
    name = "license";
    builder.ruby = ./license_builder.rb;
    inherit src;
    commit = builtins.getEnv "commit";
    nixcrpkgs_commit = builtins.getEnv "nixcrpkgs_commit";
    nixpkgs_commit = builtins.getEnv "nixpkgs_commit";
    license_names = builtins.attrNames license_set;
    licenses = builtins.attrValues license_set;
  };

  installer = env.make_derivation {
    name = "${config_name}-installer";
    builder.ruby =
      if env.os == "windows" then ./windows_installer_builder.rb
      else if env.os == "linux" then ./linux_installer_builder.rb
      else if env.os == "macos" then ./macos_installer_builder.rb
      else throw "?";
    inherit src config_name payload license;
    libusbp = env.libusbp;
  };

in
payload // { inherit env license installer; }
> nix build '.#win32'
warning: Git tree '/home/sean/ommit/ommit' is dirty
error: builder for '/nix/store/h857q0jyc62x8sqpk16w161x0d3myprm-package-i686-w64-mingw32.drv' failed with exit code 2;
       last 10 log lines:
       > Build desktop app (version: 91.91.91-dev.0)...
       > /build/ommit /build
       > Info: creating stash file /build/build-ommit/.qmake.stash
       > /nix/store/jjrbagr069zpqm60yzmmab7swamg0vk0-qtbase-5.12.12-i686-w64-mingw32/bin/qmake -o Makefile.sql /nix/store/sb6m5g4hmmhg8h4k2kpvfpkb6xkgv6m1-8k2kychgbk6wvi1r15f2ha0hrd3c6cxc-source/ommit/sql.pro WITH_COVERAGE_REPORT=1
       > Project ERROR: libexslt development package not found
       > make: *** [Makefile:70: sub-sql-pro-qmake_all] Error 3
       For full logs, run 'nix log /nix/store/h857q0jyc62x8sqpk16w161x0d3myprm-package-i686-w64-mingw32.drv'.
seanybaggins commented 1 year ago 

> nix build 'nixpkgs#libxslt.dev'
> cd result-dev
> tree
.
├── bin
│   └── xslt-config
├── include
│   ├── libexslt <-------------------------- Notice libexslt here
│   │   ├── exsltconfig.h
│   │   ├── exsltexports.h
│   │   └── exslt.h
│   └── libxslt
│       ├── attributes.h
│       ├── documents.h
│       ├── extensions.h
│       ├── extra.h
│       ├── functions.h
│       ├── imports.h
│       ├── keys.h
│       ├── namespaces.h
│       ├── numbersInternals.h
│       ├── pattern.h
│       ├── preproc.h
│       ├── security.h
│       ├── templates.h
│       ├── transform.h
│       ├── variables.h
│       ├── xsltconfig.h
│       ├── xsltexports.h
│       ├── xslt.h
│       ├── xsltInternals.h
│       ├── xsltlocale.h
│       └── xsltutils.h
├── lib
│   ├── cmake
│   │   └── libxslt
│   │       ├── FindGcrypt.cmake
│   │       └── libxslt-config.cmake
│   ├── pkgconfig
│   │   ├── libexslt.pc <--------------- Notice libexslt here.
│   │   └── libxslt.pc
│   └── xsltConf.sh
├── nix-support
│   └── propagated-build-inputs
└── share
    └── aclocal
        └── libxslt.m4

11 directories, 32 files
DavidEGrayson commented 1 year ago

I don't know what code you are using to find libexslt: your bug reports should include all the details necessary to reproduce the bug without us having to think too hard about it.

Also, note that anything you get from nixpkgs is going to be built for your native system by the nixpkgs compiler toolchain, so it's not usually going to work to put it into cross_inputs, which is supposed to be a list of cross-compiled dependencies built by the nixcrpkgs toolchian.

seanybaggins commented 1 year ago

Sorry. I am somewhat limited what I can share given I am working on proprietary code. I like that you managed to get qt cross compiling for windows. Huge props there. I might try to port your qt packages just as overlays within nixpkgs. That or trying to port my current overlays to nixcrpkgs.

DavidEGrayson commented 1 year ago

I'm glad you are enjoying the project! I do recommend using Qt 6 (env.qt6) if you can: it took much less effort for me to port it, and I didn't have to resort to generating my own CMake files with Ruby like I did with Qt 5.

It looks like you're using QMake, so if you keep having trouble you could just make a minimal qmake file with 5-10 lines of code in it that produces the same error message, and post it along with the Nix file. Then I'd be able to reproduce the issue and might be able to help.

seanybaggins commented 1 year ago

So my knowledge of Qt is somewhat limited, but from what I can see in the MyprojectPro directory, I have a common.pri file with the following line PKGCONFIG += libexslt. I think that is the crux of the issue.

common.pri

DEFINES += USE_BOOST_REGEX
DEFINES += USE_SSL
DEFINES += XMLSEC_CRYPTO_OPENSSL
DEFINES += BOOST_ENABLE_ASSERT_HANDLER
DEFINES += BOOST_LOG_DYN_LINK

QMAKE_CXXFLAGS += -g
QMAKE_LFLAGS_RELEASE =

INCLUDEPATH += "$${PWD}/../Src"
INCLUDEPATH += $${OUT_PWD}/ui

DEPENDPATH += "$${PWD}/../Src"

UI_DIR = $${OUT_PWD}/ui
MOC_DIR = $${OUT_PWD}/$${TARGET}_moc
OBJECTS_DIR = $${OUT_PWD}/$${TARGET}_obj
RCC_DIR = $${OUT_PWD}/$${TARGET}_rcc

QT       += core gui multimedia widgets

CONFIG += object_parallel_to_source
CONFIG -= depend_includepath
CONFIG += c++17
CONFIG += link_pkgconfig

PKGCONFIG += libcurl
PKGCONFIG += libexslt
PKGCONFIG += libjpeg
PKGCONFIG += lua
PKGCONFIG += libpng16
PKGCONFIG += libusb-1.0
PKGCONFIG += libxml-2.0
PKGCONFIG += sqlcipher
PKGCONFIG += freetype2
PKGCONFIG += xmlsec1-openssl
PKGCONFIG += zlib

LIBS += -lboost_chrono
LIBS += -lboost_date_time
LIBS += -lboost_filesystem
LIBS += -lboost_log
LIBS += -lboost_log_setup
LIBS += -lboost_program_options
LIBS += -lboost_regex
LIBS += -lboost_system
LIBS += -lboost_thread
LIBS += -lboost_iostreams

defined(WITH_COVERAGE_REPORT, var): QMAKE_CXXFLAGS += --coverage
defined(WITH_COVERAGE_REPORT, var): LIBS += --coverage

win32 {
    LIB_ROOT_DIR = $$shell_path($${MINGW_PATH})
    QMAKE_PKG_CONFIG=pkg-config

    PKGCONFIG += libcrypto
    PKGCONFIG += libssl
    PKGCONFIG += libxslt

    DEFINES += USE_BONJOUR_DDNS

    QMAKE_CXXFLAGS += -march=i686 -finline-functions
    QMAKE_CXXFLAGS += "-DBOOST_BIND_ENABLE_STDCALL"
    QMAKE_CXXFLAGS += -isystem $$shell_path("$$(MINGW_PATH)/include")

    LIBS += -L$$(MINGW_PATH)/lib
    LIBS += -L$$(MINGW_PATH)/bin
    LIBS += -lws2_32
    LIBS += -lwsock32
    LIBS += -lMightexBridge
    LIBS += -lcyapi
    LIBS += -lsetupapi
    LIBS += -lnicaiu
    LIBS += -ldnssd
    LIBS += -lstdc++
}

linux {
    LIB_ROOT_DIR = $$shell_path($$GNU_PATH)

    PKGCONFIG += libudev
    PKGCONFIG += avahi-client

    LIBS += -ldl

    DEFINES += USE_AVAHI_DDNS
}

sql.pro

TEMPLATE = lib
CONFIG += staticlib
TARGET = LdsSql
DESTDIR = ./LdsLibs

include (common.pri)
include (sql.pri)

someProject.pro

app.file = someProj.pro
app.depends += libsql
app.depends += libxml

test.file = someProjTest.pro
test.depends += libsql
test.depends += libxml

libsql.file = sql.pro
libxml.file = xml.pro
DavidEGrayson commented 1 year ago

Don't forget my previous comment:

Also, note that anything you get from nixpkgs is going to be built for your native system by the nixpkgs compiler toolchain, so it's not usually going to work to put it into cross_inputs, which is supposed to be a list of cross-compiled dependencies built by the nixcrpkgs toolchian.

Separately, I chose not to override the default behavior of pkg-config, so by default it only finds native libraries and can still be used to build native software. If you want to find cross-compiled libraries you need to use the wrapper provided by nixcrpkgs named pkg-config-cross (or set your environment variables properly). You neglected to show how you are invoking qmake but I'm guessing you forgot this. This file has useful info about the wrappers provided by nixcrpkgs:

https://github.com/pololu/nixcrpkgs/blob/cd1860e/wrappers/builder.sh

Also, I don't think I did much work QMake and Qt 5. I used CMake for my applications, and I think I used only a small subset of the features provided by the Qt project's CMake files. So you might run into more problems along the way.

seanybaggins commented 1 year ago

Trying this a little differently... Using nixcrpkgs as a flake input. I have listed your qt package as one of the native build inputs. Seem to be running into a similar issue where qmake does not know where to find the libraries.

Here is how qmake is being invoked

    qmake WITH_COVERAGE_REPORT=1 -o Makefile $ProjectFile && \
    make qmake_all

Here is my flake

{
  description = "A flake for somePackage development and build environment";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/5e871d8aa6f57cc8e0dc087d1c5013f6e212b4ce";
    nixcrpkgs.url = "github:seanybaggins/nixcrpkgs";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, utils, nixcrpkgs }:
    utils.lib.eachSystem [ "x86_64-linux" ] (system:
      let
        overlays = [ (import ./nix/overlays) ];
        pkgs = import nixpkgs
          {
            inherit system overlays;
            config = {
              permittedInsecurePackages = [
                "python-2.7.18.6"
              ];
            };
          };
        crpkgs = import nixcrpkgs { nixpkgs = pkgs; };
        mkBuildInputs = pkgsHostPlatform:
          with pkgsHostPlatform; [
            crpkgs.win32.qt
            curl.dev
            crpkgs.win32.libusb
            boost171
          ];
        mkNativeBuildInputs = pkgsBuildPlatform: pkgsHostPlatform:
          (with pkgsBuildPlatform; [
            bear
            git
            crpkgs.win32.qt
          ]) ++
          (with pkgsHostPlatform; [
            pkg-config
          ]);

        mkSomePackageDerivation = pkgsBuildPlatform: pkgsHostPlatform:
          pkgsHostPlatform.stdenv.mkDerivation rec {
            pname = "somePackage";
            version = "0.0.1";
            src = ./.;
            dontWrapQtApps = "1";
            buildInputs = mkBuildInputs pkgsHostPlatform;
            nativeBuildInputs = mkNativeBuildInputs pkgsBuildPlatform pkgsHostPlatform;
            QMAKE_CFLAGS = "-I${pkgsHostPlatform.curl.dev}/include";
            QMAKE_CXXFLAGS = "-I${pkgsHostPlatform.curl.dev}/include";
            QMAKE_LFLAGS = "-L${pkgsHostPlatform.curl.dev}/lib";
            PKG_CONFIG_PATH = "${pkgsHostPlatform.curl.dev}/lib/pkgconfig";
            PKG_CONFIG_LIBDIR = "${pkgsHostPlatform.curl.dev}/lib/pkgconfig";
            PKGCONFIG = "${pkgsHostPlatform.curl.dev}/lib/pkgconfig";
            preConfigure = ''
              export PKG_CONFIG_PATH="${pkgs.lib.makeSearchPath "/lib/pkgconfig" buildInputs}:$PKG_CONFIG_PATH"
            '';
            configurePhase = ''
              echo "Nix package manager here. configurePhase"
              echo "Nothing to configure"
            '';

            buildPhase = ''
              echo "Nix package manager here. buildPhase"
              bash ./Scripts/builder.sh somePackage
            '';

            installPhase = ''
              mkdir -p $out/bin
              cp -r ./build-somePackage/App/* $out/bin
            '';

          };

      in
      {
        packages.somePackage-mingw32 = mkSomePackageDerivation pkgs pkgs.pkgsCross.mingw32;
      }
    );
}

All of my attempts to add environment variables within the derivation appear to have no affect.

nix build '.#somepackage-mingw32'
warning: Git tree '/home/sean/somepackage/somepackage-nixcrpkgs' is dirty
error: builder for '/nix/store/xms9pvql8cx3j26nsrrynadvdmyiwagm-somepackage-i686-w64-mingw32-0.0.1.drv' failed with exit code 5;
       last 10 log lines:
       > nix
       > tools
       > Build desktop app (version: 91.91.91-dev.0)...
       > Sean Link
       > /build/vv4z997akl26pq66m2i2yambbxi74iwx-source/build-somepackage /build/vv4z997akl26pq66m2i2yambbxi74iwx-source
       > Info: creating stash file /build/vv4z997akl26pq66m2i2yambbxi74iwx-source/build-somepackage/.qmake.stash
       > /nix/store/jjrbagr069zpqm60yzmmab7swamg0vk0-qtbase-5.12.12-i686-w64-mingw32/bin/qmake -o Makefile.sql /nix/store/a31avabggbq0pfc8yja6y79r2vqdga7m-vv4z997akl26pq66m2i2yambbxi74iwx-source/somepackageProj/sql.pro WITH_COVERAGE_REPORT=1
       > Project ERROR: libcurl development package not found
       > make: *** [Makefile:70: sub-sql-pro-qmake_all] Error 3
       > ERROR: Unable to create qmake files
       For full logs, run 'nix log /nix/store/xms9pvql8cx3j26nsrrynadvdmyiwagm-somepackage-i686-w64-mingw32-0.0.1.drv'.
DavidEGrayson commented 1 year ago

Maybe qmake is setting PKG_CONFIG_PATH somewhere and overriding what you're trying to do. Also, nixpkgs itself probably has setup scripts written in Bash which override it. You should at least print out the values of any environment variables you care about right before you run qmake just to make sure they are OK.

Sometimes in the past I actually have overridden pkg-config with my own wrapper script written in Ruby that logs all its arguments and environment variables somewhere, so I can analyze it later and see how it was invoked. I don't know if I saved that code anywhere, I can't find it.

P.S. I don't know much about flakes and I see you are still liberally mixing together native software and stuff that was cross-compiled with nixcrpkgs, which is likely to cause big problems.

seanybaggins commented 1 year ago

P.S. I don't know much about flakes and I see you are still liberally mixing together native software and stuff that was cross-compiled with nixcrpkgs, which is likely to cause big problems.

Oh yeah. crpkgs.win32.qt in nativeBuildInputs looks like a huge no no. Changed it to crpkgs.linux-x86.qt. Trying to build now. Stand by.

seanybaggins commented 1 year ago

I think what I originally had was better. 

nix build '.#somepackage-mingw32'
warning: Git tree '/home/sean/lightdeck/somepackage-nixcrpkgs' is dirty
error: builder for '/nix/store/6lj93dv65vvwv57i0jzbcddak7jlhjkj-somepackage-i686-w64-mingw32-0.0.1.drv' failed with exit code 5;
       last 10 log lines:
       > nix
       > tools
       > Build desktop app (version: 91.91.91-dev.0)...
       > Sean Link
       > /build/ng0dh1c643hnafh34hwx9pa6p9b0fw94-source/build-somepackage /build/ng0dh1c643hnafh34hwx9pa6p9b0fw94-source
       > Project ERROR: Cannot run target compiler 'i686-linux-musleabi-g++'. Output:
       > ===================
       > ===================
       > Maybe you forgot to setup the environment?
       > ERROR: Unable to create qmake files
       For full logs, run 'nix log /nix/store/6lj93dv65vvwv57i0jzbcddak7jlhjkj-somepackage-i686-w64-mingw32-0.0.1.drv'.

Oddly enough, this is the same error I get when trying to use qmake in nixpkgs. I guess what my question is more generally is how do you use a cross compiled package from nixpkgs when it does not exist in nixcrpkgs?

DavidEGrayson commented 1 year ago

You can add the package in your fork of nixcrpkgs the same way other packages are added. Or if you want the code to live outside of nixcrpkgs, when you evaluate nixcrpkgs, it returns a set and you can access the win32.make_derivation attribute of that set to get a function that helps you cross-compile the software.

DavidEGrayson commented 1 year ago

Oh, if you actually did succeed in cross-compiling a package with Nixpkgs and you want to use it with nixcrpkgs you could try putting it in the cross_inputs argument to make_derivation but I still imagine there will be lots of issues due to differences in the toolchain, and use of dynamic libraries in Nixpkgs, and more. Mixing binaries from different toolchains is just risky unless you really know what you're doing.

nixcrpkgs was designed to be its own set of packages that work well with each other and there hasn't been any attempt to make it interoperate with cross-compiled stuff from nixpkgs, so if you go that route you need to know that you are inventing something new and you will probably need to be a power user of the Nix language, the Nixpkgs code base, build systems, and compilers. The cross-compilation in Nixpkgs was pretty broken and/or lacking when I made nixcrpkgs and I'm not sure how much it has advanced.