search

ganeti / ganeti

Ganeti is a virtual machine cluster management tool built on top of existing virtualization technologies such as Xen or KVM and other open source software.
http://www.ganeti.org
BSD 2-Clause "Simplified" License
513 stars 110 forks source link

build Haskell part with Cabal #1773

Closed thielema closed 3 months ago

thielema commented 3 months ago

ignore it for now, I am just abusing GitHub CI for running clean builds

saschalucas commented 3 months ago

ignore it for now, I am just abusing GitHub CI for running clean builds

OK. Probably it is possible to enable github actions under settings in your ganeti fork? Then the CI-Matrix tests would run on push to your branches without the need for a PR...

thielema commented 3 months ago

I have activated something in my fork, but CI is not yet running. I have to investigate. Closing this pull request for now.

thielema commented 3 months ago

CI actions are still not run in my fork. I guess this has to do with the configuration

on:
  push:
    branches:
      - master

in .github/workflows/ci.yml. Of course, I want to run actions on my feature branches. So I would have to change ci.yml. But then this would become part of a later pull request. Does not sound sensible. It does not seem right to me, that this configuration is part of the Git repository, at all.

thielema commented 3 months ago

Ok, I am now doing a pull-request to myself. This triggers the CI scripts.

thielema commented 3 months ago

The executables htools and mon-collector are also symbolically linked to new names hpc-htools and hpc-mon-collector. Why two names for the same executable and could we stick to one name? autotools/build-bash-completion seems to expect htools. Some tools are also in different directories, namely src/, apps/, test/hs. What is the purpose of this duplication?

thielema commented 3 months ago

The CI scripts now succeed with the Cabal build. However, I still have to symbolically link executables, e.g. from dist/build/htools/htools to src/htools, because build-bash-completion does not find it in dist/build/htools/htools, even when the full path is given. That's pretty strange.

saschalucas commented 3 months ago

The CI scripts now succeed with the Cabal build.

That's great. Please don't forget to reopen this PR if you feel it's ready.

However, I still have to symbolically link executables, e.g. from dist/build/htools/htools to src/htools, because build-bash-completion does not find it in dist/build/htools/htools, even when the full path is given. That's pretty strange.

Yes, strange. Don't know if it has something to do with autotools/run-in-tempdir?

thielema commented 3 months ago

The executables htools and mon-collector are also symbolically linked to new names hpc-htools and hpc-mon-collector. Why two names for the same executable and could we stick to one name?

There is Makefile.live-test. It seems hpc refers to the GHC tool for profiling and coverage. For profiling, programs must be compiled with special options, thus a separate executable would make sense. But then htools it should be moved and not linked to test/hs.

thielema commented 3 months ago

I am trying to run make dist (both in my cabal-build branch and in master) and it builds the Haskell executables. This does not look right. But then make dist in master fails with "fatal: No names found, cannot describe anything."

saschalucas commented 3 months ago

I am trying to run make dist (both in my cabal-build branch and in master) and it builds the Haskell executables. This does not look right. But then make dist in master fails with "fatal: No names found, cannot describe anything."

This looks like a git error message. Makefile.am has 2 invocations of git describe to somehow discover a version (tag).

thielema commented 3 months ago

I re-open the pull request with updated commits.

What I have done: Configure, build and generate Haskell library documentation (Haddock) via "Cabal, the library", that is, with runhaskell Setup, that is, no cabal-install needed and building completely offline.

Haskell source files for executables are moved permanently to app directory and not only linked temporarily anymore. The reason for the separation is that ghc --make compiles all reachable modules and only import precompiled modules if they are not reachable. With the library source files in src and the executable source files in the app directory, GHC compiles the library part of Ganeti once and links it to every executable. This is standard practice.

Executables are no longer put in src but are now linked to an exe directory. This way, src stays clean and is reserved for actual text files. I tried to avoid the temporary exe directory by just using Cabal's dist/build. However, some executables are not Haskell binaries. I could further refine the directory set and create an autogen directory for autogenerated modules. Or I could use Cabal preprocessors and locate autogenerated modules in dist/build instead. I will postpone this issue for now.

The current state is: [ ] The CI tests at GitHub succeed. [ ] Building on your servers works. [ ] make clean succeeds. [ ] make install installs the same number of directories and files as before. [ ] make dist succeeds. [ ] Building from the source tarball (as generated before) succeeds and number of installed files after make install is still the same.

Please tell me, if more tests are necessary for you.

The Automake setup was convoluted and is still convoluted. Paths are constructed, collected and then filtered and dissected again. This is in parts dictated or at least encouraged by Automake. It is hard to track where and what part of a path is eventually used where. It is likely that I have broken something. E.g. I have not tested the profiling&coverage stuff.

I am afraid the build system is hard to get right with Automake and we should not try too hard.

My plan is the following: Leave things as they are now and if time permits, start to write a new alternative build system that uses standard Haskell and Python packaging systems, that is, Cabal and PIP. Cabal already provides configuring, building, documenting, installing, create source tarballs, running preprocessor and autogenerate modules. Then we put some stuff around it in order to glue the Haskell and the Python part, maybe with hand-written configure files or with Just (aka Justfiles) or Autodafe. Then watch what build functionality you actually need today. When the new build system becomes mature, drop the Automake stuff.

If you decide to merge, please keep the single commits intact. I think we will need it later to git-bisect precisely to were things might have been broken.

thielema commented 3 months ago

re-open, however the source branch is not updated here so far

I don't know why GitHub does not update the branch. Thus I ask you to consider the current branch state here: https://github.com/thielema/ganeti/tree/cabal-build I still have to rebase to new commits in master.

saschalucas commented 3 months ago

I don't know why GitHub does not update the branch.

Looks like a dead end here, Repoen is grayed out with the comment "cabal-build branch was force-pushed or recreated". My be the best way is to open a new PR.

thielema commented 3 months ago

On Fri, 9 Aug 2024, saschalucas wrote:

  I don't know why GitHub does not update the branch.

Looks like a dead end here, Repoen is grayed out with the comment "cabal-build branch was force-pushed or recreated". My be the best way is to open a new PR.

I think the pull-request at ganeti main repository is not a necessary condition to merge the commits into master. You can just pull in this one:

https://github.com/thielema/ganeti/pull/1

(And please, keep the commits intact if you decide to merge.)

saschalucas commented 3 months ago

I think the pull-request at ganeti main repository is not a necessary condition to merge the commits into master. You can just pull in this one: thielema#1 (And please, keep the commits intact if you decide to merge.)

Sure, plain git is enough. Using PRs it's possible for others to participate (in good old days one would send patches trough a mailing list). Therefore I've create PR #1785.