Cabal unable to find headers generated by FFI

Describe the bug Cabal seems unable to figure where the .h header files that GHC generated on its behalf, are placed within the dist-newstyle/ path.

To Reproduce

Try to clone and build this sample project: https://github.com/mouse07410/h-tst2.git It should work:

$ cabal build
Resolving dependencies...
Build profile: -w ghc-8.10.2 -O1
In order, the following will be built (use -v for more details):
 - h-tst2-0.1.0.0 (lib) (first run)
 - h-tst2-0.1.0.0 (exe:test) (first run)
 - h-tst2-0.1.0.0 (exe:h-tst2) (first run)
Configuring library for h-tst2-0.1.0.0..
Preprocessing library for h-tst2-0.1.0.0..
Building library for h-tst2-0.1.0.0..
[1 of 2] Compiling Fib              ( src/Fib.hs, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build/Fib.o, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build/Fib.dyn_o )
[2 of 2] Compiling Paths_h_tst2     ( /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build/autogen/Paths_h_tst2.hs, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build/Paths_h_tst2.o, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build/Paths_h_tst2.dyn_o )
Configuring executable 'test' for h-tst2-0.1.0.0..
Configuring executable 'h-tst2' for h-tst2-0.1.0.0..
Warning: The package has an extraneous version range for a dependency on an
internal library: h-tst2 -any && ==0.1.0.0, h-tst2 -any && ==0.1.0.0. This
version range includes the current package but isn't needed as the current
package's library will always be used.
Warning: The package has an extraneous version range for a dependency on an
internal library: h-tst2 -any && ==0.1.0.0, h-tst2 -any && ==0.1.0.0. This
version range includes the current package but isn't needed as the current
package's library will always be used.
Preprocessing executable 'test' for h-tst2-0.1.0.0..
Preprocessing executable 'h-tst2' for h-tst2-0.1.0.0..
Building executable 'test' for h-tst2-0.1.0.0..
Building executable 'h-tst2' for h-tst2-0.1.0.0..
[1 of 2] Compiling Main             ( app/Main.hs, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/x/h-tst2/build/h-tst2/h-tst2-tmp/Main.dyn_o )
[2 of 2] Compiling Paths_h_tst2     ( /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/x/h-tst2/build/h-tst2/autogen/Paths_h_tst2.hs, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/x/h-tst2/build/h-tst2/h-tst2-tmp/Paths_h_tst2.dyn_o )
Linking /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/x/h-tst2/build/h-tst2/h-tst2 ...
$ cabal run test
Up to date
Fibonacci: 267914296

Then comment out this obviously-wrong kludge-y line from h-tst2.cabal:

executable test
    include-dirs:
        c_code
    --  , dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build
    main-is:           test.c
    ghc-options:       -no-hs-main
    build-depends:
        base >=4.13.0.0
      , h-tst2
    default-language:  Haskell2010

do cabal clean and try the build again - the build would fail:

$ cabal clean
$ cabal build
Resolving dependencies...
Build profile: -w ghc-8.10.2 -O1
In order, the following will be built (use -v for more details):
 - h-tst2-0.1.0.0 (lib) (first run)
 - h-tst2-0.1.0.0 (exe:test) (first run)
 - h-tst2-0.1.0.0 (exe:h-tst2) (first run)
Configuring library for h-tst2-0.1.0.0..
Preprocessing library for h-tst2-0.1.0.0..
Building library for h-tst2-0.1.0.0..
[1 of 2] Compiling Fib              ( src/Fib.hs, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build/Fib.o, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build/Fib.dyn_o )
[2 of 2] Compiling Paths_h_tst2     ( /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build/autogen/Paths_h_tst2.hs, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build/Paths_h_tst2.o, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build/Paths_h_tst2.dyn_o )
Configuring executable 'test' for h-tst2-0.1.0.0..
Configuring executable 'h-tst2' for h-tst2-0.1.0.0..
Warning: The package has an extraneous version range for a dependency on an
Warning: The package has an extraneous version range for a dependency on an
internal library: h-tst2 -any && ==0.1.0.0, h-tst2 -any && ==0.1.0.0. This
internal library: h-tst2 -any && ==0.1.0.0, h-tst2 -any && ==0.1.0.0. This
version range includes the current package but isn't needed as the current
package's library will always be used.
version range includes the current package but isn't needed as the current
package's library will always be used.
Preprocessing executable 'h-tst2' for h-tst2-0.1.0.0..
Building executable 'h-tst2' for h-tst2-0.1.0.0..
Preprocessing executable 'test' for h-tst2-0.1.0.0..
Building executable 'test' for h-tst2-0.1.0.0..

test.c:5:10: error:  fatal error: 'Fib_stub.h' file not found
  |
5 | #include "Fib_stub.h"
  |          ^
#include "Fib_stub.h"
         ^~~~~~~~~~~~
1 error generated.
`clang' failed in phase `C Compiler'. (Exit code: 1)
[1 of 2] Compiling Main             ( app/Main.hs, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/x/h-tst2/build/h-tst2/h-tst2-tmp/Main.dyn_o )
[2 of 2] Compiling Paths_h_tst2     ( /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/x/h-tst2/build/h-tst2/autogen/Paths_h_tst2.hs, /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/x/h-tst2/build/h-tst2/h-tst2-tmp/Paths_h_tst2.dyn_o )
Linking /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/x/h-tst2/build/h-tst2/h-tst2 ...
cabal: Failed to build exe:test from h-tst2-0.1.0.0.

Observe that the file in question Fib_stub.h does exist:

$ fd Fib_stub.h
dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build/Fib_stub.h

As far as I understand, you are not supposed to use Foo_stub.h files directly, but copy them to known location (as Foo.h) and edit them into proper header files, to be used as part of your bigger C development.

At least when making e.g. foreign libraries, _stub.h files are something you want to distribute with your shared library binary, and that might have more stuff.

As far as I understand, you are not supposed to use Foo_stub.h files directly,

I never heard that.

...but copy them to a known location (as Foo.h)

Re. renaming - probably depends. With other ecosystems I've seen it both ways - using Foo_stub.h and using Foo.h, without clear definition of reasons why one or the other.

Re. copying - sure. But how do I do it within the Cabal "realm"?

and edit them into proper header files,

Honestly, in general I don't see the need - these generated _stub.h files appear proper, and perfectly usable as-is.

to be used as part of your bigger C development

Sure. As I expect that more often than not a (shared) library would be installed (copied to) somewhere, and other software on that machine would link against it, and load it. Same for the header files.

But in this toy project, the library can stay where it's built - Cabal and GHC seem to manage finding it without any problem.

It's only this one generated header file that gives us trouble.

At least when making e.g. foreign libraries, _stub.h files are something you want to distribute with your shared library binary, and that might have more stuff.

Yes, certainly.

But how can we resolve this problem now? I'd be equally happy with either telling Cabal to tell GHC where to find that Fib_stub.h file, or telling Cabal to copy that file to, e.g., ./c_code/ directory after it's generated (library target built) but before the next target (executable test) starts building. Question is - how?

Basically, the problem is:

Library build creates a header file Fib_stub.h and places it inside the dist-newstyle/... Something needs to copy that file to, e.g., c_code/ directory, but Cabal doesn't do that;
Build of executable test needs Fib_stub.h, but since Cabal neither copied it to a "known" directory, nor passed it's actual location dir to GHC (who in turn did not pass it to clang), the executable build fails.

Two possible solutions:

For Cabal to copy the given file(s) to a given directory;
For Cabal to pass something like -I./dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build to clang.

(1) is required when a self-contained Haskell library is created. Users will access it via FFI, so there should be mechanisms that tell Cabal where to install these files, or how to package them.

(2) is required when one builds a mixed package that contains executables (in foreign language) which require that file to compile.

As far as I understand, the Something in

Something needs to copy that file to, ... but Cabal doesn't do that;

is a programmer.

Calling Haskell, from Haskell, but through C-wrappers feels round-about to me. I understand the idea of testing however, yet there I'd still imagine having a vanilla C-program (not compiled by Cabal) to do testing.

Yet, I have no idea how the foreign export was ever planned to be integrated into Cabal builds, so I leave this issue open for someone who knows better to comment.

As far as I understand, the Something in

Something needs to copy that file to, ... but Cabal doesn't do that;

is a programmer.

That cannot work, because you pointed out yourself that it is wrong to try to dig into the bowels of the dist-newstyle/... content. There has to be a way for Cabal to "install" those files (library itself, maybe some metadata, and header files) somewhere into a predictable stable documented location.

Calling Haskell, from Haskell, but through C-wrappers feels round-about to me

Sure, but I'm not doing that!

There's src/Fib.hs library defined in the library block of h-tst2.cabal that exposes module Fib. It provides Haskell and FFI interface to a Haskell function: Haskell function fib and FFI API to it named fib_hs. In the process of building it generates Fib_stub.h header file and libHSh-tst2-0.1.0.0-inplace-ghc8.10.2.dylib shared library itself. NB: Cabal and all the components it invokes, find that library fine.
There's a Haskell program app/Main.hs that calls fib - for no particular purpose. This is the executable h-tst2 part of the h-tst2.cabal, which is optional. It happens to work fine, but is not needed.
There's a C program test.c that calls fib_hs. For that it compiles test.c including header Fib_stub.h (which apparently it cannot find on its own, and links it with the library libHSh-tst2-0.1.0.0-inplace-ghc8.10.2.dylib, which it can find (or Cabal can find) OK.

One workaround would be to make Cabal do (1) - build the library itself, then make it take the files it generated, and install somewhere. Then, on a second invocation make Cabal build (3), taking Fib_stub.h from where the previous step installed it to.

Questions:

How can I tell Cabal where to install the library itself? This only matters for calling from the foreign stuff, because whatever Cabal itself installs into ~/.cabal/store, apparently Cabal can find, so Haskell-to-Haskell shouldn't be a problem.
How can I make the build 2-phase? So that it just builds one component, and then manually invoked to build another one?

Poking around, I found file dist-newstyle/packagedb/ghc-8.10.2/h-tst2-0.1.0.0-inplace.conf. It's content has the directories I need, if there's a way to refer to them in/from the .cabal file.

name:                 h-tst2
version:              0.1.0.0
visibility:           public
id:                   h-tst2-0.1.0.0-inplace
key:                  h-tst2-0.1.0.0-inplace
license:              MIT
copyright:            "Copyright (C) 2020, MIT"
description:
    Experiment to time performance of Haskell functions called from C

abi:                  inplace
exposed:              True
exposed-modules:      Fib
hidden-modules:       Paths_h_tst2
import-dirs:
    /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build

library-dirs:
    /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build
    /opt/local/lib/liconv /opt/local/lib /usr/local/lib

dynamic-library-dirs:
    /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/build
    /opt/local/lib/liconv /opt/local/lib /usr/local/lib

data-dir:             /Users/ur20980/src/h-tst2
hs-libraries:         HSh-tst2-0.1.0.0-inplace
include-dirs:         /opt/local/include /usr/local/include
depends:              base-4.14.1.0
haddock-interfaces:
    /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/doc/html/h-tst2/h-tst2.haddock

haddock-html:
    /Users/ur20980/src/h-tst2/dist-newstyle/build/x86_64-osx/ghc-8.10.2/h-tst2-0.1.0.0/doc/html/h-tst2

If only I could add something like include-dirs: $import-dirs, it would find the required header file.

So, is there a way to refer to one of these variables (imports-dir, libdir, dynlibdir) from inside h-tst2.cabal file, using it as the value? E.g., some way to say

. . . . .
executable test
  include-dirs:  $(import-dirs)
  main-is:           test.c
  ghc-options:       -no-hs-main
. . . . .

Note: to reproduce checkout 7052904fed8eb06b79dad25fede7fcad134548ab from the reproducer

Now Cabal works as expected - tested with 3.10.3.0 and 3.12.1.0, GHC-9.8.2 and 9.10.1 - when h-tst2.cabal specifies a directory other than . (see https://github.com/mouse07410/h-tst2/blob/master/h-tst2.cabal).

haskell / cabal

Cabal unable to find headers generated by FFI #7058