Unable to build streamly using rules_haskell/nixpkgs

(Was uncertain where to open this issue given it involves rules_{haskell,nixpkgs} and streamly.)

Currently I'm unable to bazel build @stackage//:streamly on macOS when using rules_nixpkgs haskell_register_ghc_nixpkgs and nixpkgs_cc_configure() for any recent stack_snapshot.snapshot.

The specific error is:

src/Streamly/Internal/FileSystem/Event/Darwin.hs:1086:2: error:
     warning: "Autoconf did not find the definition kFSEventStreamCreateFlagFileEvents in Darwin header files.Do you have Cocoa framework header files installed?Not compiling the Streamly.Internal.FileSystem.Event.Darwin module. Programs depending on this module may not compile. Check if HAVE_DECL_KFSEVENTSTREAMCREATEFLAGFILEEVENTS is defined in config.h generated from src/config.h.in" [-W#warnings]
     |
1086 | #warning "Autoconf did not find the definition \
     |  ^
#warning "Autoconf did not find the definition \
 ^
1 warning generated.

src/Streamly/Internal/FileSystem/Event.hs:54:51: error:
    Module
    ‘Streamly.Internal.FileSystem.Event.Darwin’
    does not export
    ‘Event’
   |
54 | import Streamly.Internal.FileSystem.Event.Darwin (Event)
   |                                                   ^^^^^

Which is a failing #if HAVE_DECL_KFSEVENTSTREAMCREATEFLAGFILEEVENTS conditional from here, which is defined in configure and config.h.in, respectively.

The streamly build instructions have a section about adding nixpkgs.darwin.apple_sdk.frameworks.Cocoa to buildInputs , additionally, streamly.cabal has frameworks: Cocoa.

It appears nixpkgs_cc_configure sets up various frameworks using -F... for the cc_wrapper, but I assume we somehow need to additionally pass the framework(s) to stackage_snapshot.extra_deps = { "streamly": [ ... ] } somehow?

If that's the case, is there some example/prior art somewhere I could crib?

Hi @brendanhay, thank you for the detailed report!

Indeed, nixpkgs_cc_configure sets up these extra search paths for the CC toolchain. Though, it looks like Cabal and the configure script don't pick that up.

I'm not familiar with Cabal's handling of framework dependencies, but it looks like it has an extra-framework-dirs field and --extra-framework-dirs flag.

Unfortunately, rules_haskell currently doesn't handle framework_includes. So, I'm not sure if it's possible to expose the search paths via extra_deps without first adding support for framework_includes. However, perhaps it is sufficient to expose the needed headers as a regular cc_library target. You could generate one using nixpkgs_package similar to here and add it to extra_deps similar to here.

If that doesn't work and one has to go through frameworks, perhaps, as a test, you could add the --extra-framework-dirs equivalent of nixpkgs_cc_configure's -F flags on the cabalopts attribute of haskell_register_ghc_nixpkgs to see if that exposes the required search paths to the configure script?

Smol update, I've tried both approaches, firstly via cc_library + extra_deps which didn't appear to work and secondly by registering GHC with hardcoded paths to the necessary existing pkgs.darwin.apple_sdk.frameworks paths via:

haskell_register_ghc_nixpkgs(
    ...
    cabalopts = [
        "--extra-framework-dirs=/nix/store/bjyn9s1s35pxdqmg6zyakd6q9wvvzv1v-apple-framework-CoreFoundation
/Library/Frameworks",
        "--extra-framework-dirs=/nix/store/q3am24bkqh0d2swbr0rdwjl0xv5sm106-apple-framework-Security/Libra
ry/Frameworks",
        "--extra-framework-dirs=/nix/store/yn5hxrvnvf5738kqgninx7555dx0h9fz-apple-framework-CoreData/Libra
ry/Frameworks",
        "--extra-framework-dirs=/nix/store/4ixb19plcksi6d2rhgn8lxs9wrzggkgv-apple-framework-CoreServices/L
ibrary/Frameworks",
        "--extra-framework-dirs=/nix/store/pl6fhk77yybaj710qj0v6vpf108cb6p9-apple-framework-Foundation/Lib
rary/Frameworks",
    ]
)

(There's not some form of Bazel hermeticity that would render that pointless?)

Neither appeared to work and I could see the configure script failing to recognise any of the kFSEventStream.. declarations:

checking whether kFSEventStreamCreateFlagFileEvents is declared... no
checking whether kFSEventStreamCreateFlagFullHistory is declared... no
checking whether kFSEventStreamEventFlagItemCloned is declared... no
checking whether kFSEventStreamEventFlagItemIsHardlink is declared... no

But I'm rather more confident in my confusion about how macOS frameworks + includes work than any failings in your suggested approaches. (Basically only what's documented via https://gitlab.haskell.org/ghc/ghc/-/wikis/osx-frameworks)

I also tried various buildEnv approaches to try and create a more suitable collection of framework paths, but you end up needing to refer to directories specifically in {setup,}_deps via $(location ...) to try and use either GHC's -framework-path or Cabal's --extra-framework-dirs=.

I'm certain someone else with macOS familiarity could probably wrangle this but for myself removing the dependency is probably going to be the easiest route.

For posterity here's the more recent cc_library approach I tried, but there have been (many) others that tried various options for aligning the header/include/strip_include_prefix paths:

# WORKSPACE

FRAMEWORKS = [
    "CoreServices",
    "CoreFoundation",
    "Security",
    "Foundation",
    "Cocoa",
]

[
  nixpkgs_package(
    name = "nixpkgs_%s" % name,
    attribute_path = "darwin.apple_sdk.frameworks.%s" % name,
    repository = "@nixpkgs",
    build_file_content = """
package(default_visibility = [ "//visibility:public" ])

filegroup(
  name = "include",
  srcs = glob(["Library/Frameworks/{name}.framework/Headers/*.h"])
)

cc_library(
  name = "nixpkgs_{name}",
  hdrs = glob(["Library/Frameworks/{name}.framework/Headers/*.h"]),
  strip_include_prefix = "Library/Frameworks"
)
""".format(name = name)
  ) for name in FRAMEWORKS
]

And then the usual:

stack_snapshot(
   ...
  extra_deps = {
    "streamly": [
      "@nixpkgs_{name}//:nixpkgs_{name}".format(name = name)
      for name in FRAMEWORKS
    ],
)

Is there any direct way to debug that such an artefact contains/exposes the headers, just to verify?

@brendanhay Thank you for sharing the steps taken.

Is there any direct way to debug that such an artefact contains/exposes the headers, just to verify?

Perhaps Starlark Queries could help. You can use them to inspect the target's CcInfo provider, drilling down on compilation_context for headers and linking_context for libraries. r2r-dev/bazel-cquery-debugging might be worth a look for some ready made debug info and for inspiration.

But I'm rather more confident in my confusion about how macOS frameworks + includes work than any failings in your suggested approaches. (Basically only what's documented via https://gitlab.haskell.org/ghc/ghc/-/wikis/osx-frameworks)

I feel similarly confused about them, looking at the "more-unix-y-way", the -I or cc_library(hdrs = ...) route should work.

tweag / rules_haskell

Unable to build streamly using rules_haskell/nixpkgs #1698