search

haskell / ghcup-hs

https://www.haskell.org/ghcup/
GNU Lesser General Public License v3.0
279 stars 83 forks source link

Improve 'ghcup compile ghc' interface #846

Open hasufell opened 1 year ago

hasufell commented 1 year ago

The main issues are:

  1. passing additional flags to build configure...
  2. but also passing additional flags to bindist configure
  3. and also passing additional flags to hadrian itself

Currently we support the first case via -- longopts. But that can only cover one use case.

This is particularly important for cross builds, which may need different than default settings.

@hsyl20

hasufell commented 1 year ago

Should probably also make hadrian default or detect whether a make build system exists. --hadrian flag should only be required if there's both make and hadrian.

chreekat commented 1 year ago

Should probably also make hadrian default or detect whether a make build system exists. --hadrian flag should only be required if there's both make and hadrian.

Is it possible to say "Use make for <9.6 and Hadrian for >= 9.6"?

And fwiw I think 9.4 was the only series that officially supported both, although I also think Hadrian was pretty immature then.

hasufell commented 1 year ago

Is it possible to say "Use make for <9.6 and Hadrian for >= 9.6"?

Currently, my idea is just to look for the hadrian files and if they're not there, assume make:

https://github.com/haskell/ghcup-hs/pull/854/files#diff-3b9b75e85b8e396244d24bf04ef933cae33b4d20df6f5889304bfb36696c43d6R957

The choice can be made explicit with --hadrian | --make.

chreekat commented 1 year ago

I don't know that it's valuable to make it an option to users if 9.4 is the only version that officially supported both. <9.4, --hadrian might work but will be buggy (and the bugs will never get fixed); >=9.6, --make will fail.

hasufell commented 1 year ago

Yes, but GHC has made it extra hard to detect which version you're actually compiling: https://gitlab.haskell.org/ghc/ghc/-/issues/22322

The fix was backported, but I have no clear overview at the moment what versions work and which don't.

As such I'm moving away from logic that depends on version numbers. If you try to compile 9.6.1 with make, that may fail. But maybe not, because you're using custom patched bindists, a different branch etc (remember you can already supply patches before the build... that's explicitly supported).

Dynamic detection seems more flexible.

What we probably want is a more descriptive error in case you try to build 9.6.1 with make: "The make build system has been removed since 9.6. You probably want to omit '--make'."

hasufell commented 1 year ago

And I'll probably support the VERSION file, in case it exists.

hasufell commented 1 year ago

What's missing now is the question on how to support additional arguments to ./configure and hadrian ....

If we compile, we have two configure phases:

  1. configure before compilation
  2. configure of the bindist

It's not clear whether they support the exact same arguments. Or if you'd even want that.

For hadrian I think we're mostly good with the --flavour=foo switch, since many things are now flavour transformers.

0rphee commented 1 year ago

EDIT: Sorry, i was running this with github actions and it seems github actions doesn't display a part of the logs. I'll look into it

Hi, thanks again for so awesome tool, and your time.

(i think my issues are related to this ticket) I'm trying to build a cross-compiler from host x86_64-macos to target aarch64-macos with ghcup compile ghc

I've tried multiple combinations of ghc target versions (8.10.7, 9.0.2, 9.2.8, 9.4.5, 9.6.2) (all with bootstrapping version 9.4.5), and additional flags (particularly --jobs), without any success. I am mostly asking if I should keep trying to build using ghcup or I just should try directly from ghc source.

Here I leave some logs w/errors while trying to compile. ghcup compile ghc -v 9.6.2 -b 9.4.5 "" -x aarch64-apple-darwin with [ Error ] []8;;https://errors.haskell.org/messages/GHCup-00841\GHCup-00841]8;;\] Process "make" with arguments [] failed with exit code 2. ghcup compile ghc -v 9.2.8 -b 9.4.5 "" -x aarch64-apple-darwin with 

[ Error ] []8;;https://errors.haskell.org/messages/GHCup-00841\GHCup-00841]8;;\] Process "sh" with arguments ["./configure",
[ ...   ]                              "--target=aarch64-apple-darwin",
[ ...   ]                              "--prefix=/Users/runner/.ghcup/ghc/aarch64-apple-darwin-9.2.8",
[ ...   ]                              ""] failed with exit code 1.

ghcup compile ghc -v 9.0.2 -b 9.4.5 "" -x aarch64-apple-darwin with

[ Error ] []8;;https://errors.haskell.org/messages/GHCup-00841\GHCup-00841]8;;\] Process "sh" with arguments ["./configure",
[ ...   ]                              "--target=aarch64-apple-darwin",
[ ...   ]                              "--prefix=/Users/runner/.ghcup/ghc/aarch64-apple-darwin-9.0.2",
[ ...   ]                              ""] failed with exit code 1.

ghcup compile ghc -v 8.10.7 -b 9.4.5 "" -x aarch64-apple-darwin with

[ Error ] []8;;https://errors.haskell.org/messages/GHCup-00841\GHCup-00841]8;;\] Process "sh" with arguments ["./configure",
[ ...   ]                              "--target=aarch64-apple-darwin",
[ ...   ]                              "--prefix=/Users/runner/.ghcup/ghc/aarch64-apple-darwin-8.10.7",
[ ...   ]                              ""] failed with exit code 1.

The error 00841 is: 

A process exited prematurely [GHCup-00841]

This error can happen for a variety of reasons and may indicate that GHCup needs to improve error handling/messages. Consider [raising an issue](https://github.com/haskell/ghcup-hs/issues).

One common source of errors is a missing or non-working C toolchain, leading to errors when trying to install GHC:

(i didn't post examples with -j flag, but the error was something like "make no parse -j8". if its necessary ill look it up

dfordivam commented 3 months ago

Regarding the hadrian flags, I figured out that the "binary-dist" target does work with the '--docs=none'.

We already have a way to specify -j and --flavour flags.

In addition the user might want to specify these

--docs[=TARGET]             Strip down docs targets (none, no-haddocks, no-sphinx[-{html, pdfs, man}].
--bignum[=BACKEND]          Select ghc-bignum backend: native, gmp (default), check-gmp, ffi.

But it would be problematic if the user specifies these

-o[BUILD_ROOT], --build-root[=BUILD_ROOT] Where to store build artifacts. (Default _build).
-c, --configure             Deprecated: Run the boot and configure scripts.

Flags other than these are likely not relevant for compilation via ghcup.

So perhaps it would be simpler if ghcup also supports --docs and --bignum flags, which are passed on to hadrian.

All hadrian flags (from ghc 9.10.1) ``` Usage: hadrian [options] [target] ... Standard options: --abbrev=FULL=SHORT Use abbreviation in status messages. --allow-redefine-rules Allow redefining built-in rules --no-allow-redefine-rules Forbid redefining built-in rules (default) --no-build Don't build anything. -C DIRECTORY, --directory=DIRECTORY Change to DIRECTORY before doing anything. --color, --colour Colorize the output. --no-color, --no-colour Don't colorize the output. --compact[=yes|no|auto] Use a compact Bazel/Buck style output. -d[FILE], --debug[=FILE] Print lots of debugging information. --demo Run in demo mode. --digest Files change when digest changes. --digest-and Files change when modtime and digest change. --digest-and-input Files change on modtime (and digest for inputs). --digest-or Files change when modtime or digest change. --digest-not Files change when modtime changes. --exception Throw exceptions directly. --flush=N Flush metadata every N seconds. --never-flush Never explicitly flush metadata. -h, --help Print this message and exit. -j[N], --jobs[=N] Allow N jobs/threads at once [default CPUs]. --keep-going Keep going when some targets can't be made. -l, --lint Perform limited validation after the run. --lint-watch=PATTERN Error if any of the patterns are created (expensive). --lint-fsatrace[=DIR] Use fsatrace to do validation [in current dir]. --lint-ignore=PATTERN Ignore any lint errors in these patterns. --no-lint Turn off --lint. --live[=FILE] List the files that are live [to live.txt]. -m PREFIX, --metadata=PREFIX Prefix for storing metadata files. --numeric-version Print just the version number and exit. --skip-commands Try and avoid running external programs. -B[PATTERN], --rebuild[=PATTERN] If required, these files will rebuild even if nothing has changed. --no-rebuild[=PATTERN] If required, these files will rebuild only if things have changed (default). --skip[=PATTERN] Don't rebuild matching files this run. -r[FILE], --report[=FILE], --profile[=FILE] Write out profiling information [to report.html]. --no-reports Turn off --report. --rule-version=VERSION Version of the build rules. --no-rule-version Ignore the build rules version. --share[=DIRECTORY] Shared cache location. --share-list List the shared cache files. --share-sanity Sanity check the shared cache files. --share-remove[=SUBSTRING] Remove the shared cache keys. --share-copy Copy files into the cache. --share-symlink Symlink files into the cache. -s, --silent Don't print anything. --sleep Sleep for a second before building. -S, --no-keep-going, --stop Turns off -k. --storage Write a storage log. -p[N], --progress[=N] Show progress messages [every N secs, default 5]. --no-progress Don't show progress messages. -q, --quiet Print less (pass repeatedly for even less). --no-time Don't print build time. --timings Print phase timings. -V, --verbose, --trace Print more (pass repeatedly for even more). -v, --version Print the version number and exit. -w, --print-directory Print the current directory. --no-print-directory Turn off -w, even if it was turned on implicitly. Extra options: -c, --configure Deprecated: Run the boot and configure scripts. -o BUILD_ROOT, --build-root=BUILD_ROOT Where to store build artifacts. (Default _build). --flavour[=FLAVOUR] Build flavour (Default, Devel1, Devel2, Perf, Prof, Quick or Quickest). --freeze1 Freeze Stage1 GHC. --freeze2 Freeze Stage2 GHC. --hash-unit-ids Include package hashes in unit ids. --skip-depends Skip rebuilding dependency information. --bignum[=BACKEND] Select ghc-bignum backend: native, gmp (default), check-gmp, ffi. --progress-info=STYLE Progress info style (None, Brief, Normal or Unicorn). --docs=TARGET Strip down docs targets (none, no-haddocks, no-sphinx[-{html, pdfs, man}]. -k, --keep-test-files Keep all the files generated when running the testsuite. --test-compiler=TEST_COMPILER Use given compiler [Default=stage2]. --test-config-file=CONFIG_FILE configuration file for testsuite. Default=testsuite/config/ghc --config=EXTRA_TEST_CONFIG Configurations to run test, in key=value format. --summary-junit[=TEST_SUMMARY_JUNIT] Output testsuite summary in JUnit format. --summary-metrics[=METRICS_FILE] Output testsuite performance metrics summary. --only[=TESTS] Test cases to run. --only-perf Only run performance tests. --skip-perf Skip performance tests. --test-root-dirs[=DIR1:[DIR2:...:DIRn]] Test root directories to look at (all by default). --test-speed=SPEED fast, slow or normal. Normal by default --summary[=TEST_SUMMARY] Where to output the test summary file. --test-verbose[=TEST_VERBOSE] A verbosity value between 0 and 5. 0 is silent, 4 and higher activates extra output. --test-way=TEST_WAY only run these ways --broken-test=TEST_NAME consider these tests to be broken -a, --test-accept Accept new output of tests --test-have-intree-files Run the in-tree tests even with an out of tree compiler --prefix[=PATH] Destination path for the bindist 'install' rule --complete-setting[=SETTING] Setting key to autocomplete, for the 'autocomplete' target. --haddock-for-hackage Generate documentation suitable for upload to a hackage server. ```
dfordivam commented 3 months ago

For ghc-bignum, there is one flavour transformer native_bignum https://gitlab.haskell.org/ghc/ghc/-/blob/master/hadrian/doc/flavours.md#flavour-transformers

dfordivam commented 3 months ago

The following flags are already handled by GHCup: --prefix, --target, --disable-ld-override

Flags other than these may not be necessary to be specified by most of users.

Note that for cross-compilation, most of the flags have to be specified at the build configure. ref: https://gitlab.haskell.org/ghc/ghc/-/wikis/building/cross-compiling#configuring-the-build

As I have mentioned above, and discussed here --bignum is likely not necessary, and just adding support of --docs should suffice.

Ref: All bindist configure flags (from ghc 9.10.1) ``` Usage: ./configure [OPTION]... [VAR=VALUE]... To assign environment variables (e.g., CC, CFLAGS...), specify them as VAR=VALUE. See below for descriptions of some of the useful variables. Defaults for the options are specified in brackets. Configuration: -h, --help display this help and exit --help=short display options specific to this package --help=recursive display the short help of all the included packages -V, --version display version information and exit -q, --quiet, --silent do not print `checking ...' messages --cache-file=FILE cache test results in FILE [disabled] -C, --config-cache alias for `--cache-file=config.cache' -n, --no-create do not create output files --srcdir=DIR find the sources in DIR [configure dir or `..'] Installation directories: --prefix=PREFIX install architecture-independent files in PREFIX [/usr/local] --exec-prefix=EPREFIX install architecture-dependent files in EPREFIX [PREFIX] By default, `make install' will install all the files in `/usr/local/bin', `/usr/local/lib' etc. You can specify an installation prefix other than `/usr/local' using `--prefix', for instance `--prefix=$HOME'. For better control, use the options below. Fine tuning of the installation directories: --bindir=DIR user executables [EPREFIX/bin] --sbindir=DIR system admin executables [EPREFIX/sbin] --libexecdir=DIR program executables [EPREFIX/libexec] --sysconfdir=DIR read-only single-machine data [PREFIX/etc] --sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com] --localstatedir=DIR modifiable single-machine data [PREFIX/var] --runstatedir=DIR modifiable per-process data [LOCALSTATEDIR/run] --libdir=DIR object code libraries [EPREFIX/lib] --includedir=DIR C header files [PREFIX/include] --oldincludedir=DIR C header files for non-gcc [/usr/include] --datarootdir=DIR read-only arch.-independent data root [PREFIX/share] --datadir=DIR read-only architecture-independent data [DATAROOTDIR] --infodir=DIR info documentation [DATAROOTDIR/info] --localedir=DIR locale-dependent data [DATAROOTDIR/locale] --mandir=DIR man documentation [DATAROOTDIR/man] --docdir=DIR documentation root [DATAROOTDIR/doc/ghc-9.10.1] --htmldir=DIR html documentation [DOCDIR] --dvidir=DIR dvi documentation [DOCDIR] --pdfdir=DIR pdf documentation [DOCDIR] --psdir=DIR ps documentation [DOCDIR] System types: --build=BUILD configure for building on BUILD [guessed] --host=HOST cross-compile to build programs to run on HOST [BUILD] --target=TARGET configure for building compilers for TARGET [HOST] Optional Features: --disable-option-checking ignore unrecognized --enable/--with options --disable-FEATURE do not include FEATURE (same as --enable-FEATURE=no) --enable-FEATURE[=ARG] include FEATURE [ARG=yes] --enable-distro-toolchain Do not use bundled Windows toolchain binaries. --disable-ld-override Prevent GHC from overriding the default linker used by gcc. If ld-override is enabled GHC will try to tell gcc to use whichever linker is selected by the LD environment variable. [default=override enabled] Optional Packages: --with-PACKAGE[=ARG] use PACKAGE [ARG=yes] --without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no) --with-hs-cpp=ARG Path to the Haskell (C) preprocessor for Haskell files [default=autodetect] --with-hs-cpp-flags=ARG Flags to the Haskell (C) preprocessor for Haskell files [default=autodetect] --with-cpp=ARG Path to the (C) preprocessor [default=autodetect]. If you set --with-cpp=CC, ensure -E is included in --with-cpp-flags --with-cpp-flags=ARG Flags to the (C) preprocessor [default=autodetect] Some influential environment variables: CC C compiler command CFLAGS C compiler flags LDFLAGS linker flags, e.g. -L if you have libraries in a nonstandard directory LIBS libraries to pass to the linker, e.g. -l CPPFLAGS (Objective) C/C++ preprocessor flags, e.g. -I if you have headers in a nonstandard directory CXX C++ compiler command CXXFLAGS C++ compiler flags CPP C preprocessor LLC Use as the path to LLVM's llc [default=autodetect] OPT Use as the path to LLVM's opt [default=autodetect] LLVMAS Use as the path to LLVM's assembler (typically clang) [default=autodetect] Use these variables to override the choices made by `configure' or to help it to find libraries and programs with nonstandard names/locations. ```
dfordivam commented 3 months ago

With reference to #1082, perhaps GHCup should throw an error if user specifies --config with hadrian, and tell the user to specify --make if they wish to use it. (or default to --make if it is available and user has specified --config) (done via #1085)

And as the --make has been deprecated from 9.6 onward, the only way the user would be able to do custom builds would be to specify hadrian/UserSettings.hs file as described here. Should GHCup support specifying this file via CLI option?