Open aherrmann opened 4 years ago
Ah, yeah, this is an unfortunate behavior. What I'd really like is for the allpaths
query to only follow regular deps
and not other attributes like data
or srcs
. But I don't see a way to do that with how Bazel currently works.
One possibility I've thought about would be to record more information about the graph of transitive dependencies in HaskellCompileInfo, and do the allpaths
computation in Haskell rather than via blaze cquery
.
For now, though, the approach you suggested sounds like a reasonable workaround. Would you like to send a patch?
By they way, even if we resolve this issue with genrules, you may have a problem loading everything into gghci at once; GHCi gets confused if you try to give it two source files that both have a "Main" module. In the case you mentioned, it might be fine since generate-stable-packages has a custom main_function.
One possibility I've thought about would be to record more information about the graph of transitive dependencies in HaskellCompileInfo, and do the
allpaths
computation in Haskell rather than viablaze cquery
.
That sounds interesting. Could you expand a bit on what you have in mind? I'm not sure I understand how this would work.
For now, though, the approach you suggested sounds like a reasonable workaround. Would you like to send a patch?
I've opened #11 implementing this filter.
By they way, even if we resolve this issue with genrules, you may have a problem loading everything into gghci at once; GHCi gets confused if you try to give it two source files that both have a "Main" module. In the case you mentioned, it might be fine since generate-stable-packages has a custom main_function.
That's a good point. In this particular case we do indeed get away with it. Only :main
doesn't work.
.HaskellCompile.pb: openBinaryFile: does not exist
error. However, I've noticed that two library dependencies are not picked up for some reason. I.e.$ hrepl //compiler/damlc:damlc-bootstrap //compiler/damlc/daml-lf-conversion
Fails with "could not load module" errors due to missing -package ghcide
and missing -package shake
. If I add them explicitly it works:
$ hrepl //compiler/damlc:damlc-bootstrap //compiler/damlc/daml-lf-conversion --show-commands --package @stackage//:ghcide --package @stackage//:shake
I'm not sure what is causing this. In case of ghcide
there is only one step in between damlc-bootstrap -> damlc-lib -> ghcide
. damlc-bootstrap
has no direct dependencies apart from base
and damlc-lib
, so I wonder why it's only ghcide
and shake
that are skipped.
@aherrmann shake
and ghcide
are a different rule type, haskell_cabal_library
. That's probably why your query isn't picking them up. Can you try adding that (and also haskell_toolchain_library
, I suppose) to the filter list?
One possibility I've thought about would be to record more information about the graph of transitive dependencies in HaskellCompileInfo, and do the
allpaths
computation in Haskell rather than viablaze cquery
.That sounds interesting. Could you expand a bit on what you have in mind? I'm not sure I understand how this would work.
The idea is not particularly well-formed yet, to be honest. Here's one possibility: For each Haskell rule, add a haskell_dep_info
output group that emits a new DepInfo
proto:
message DepInfo {
repeated string dep_labels = 1; // Immediate dependencies
}
Then, replace getAllIntermediateTargets
(which is currently just an allpaths($targets, $targets)
query) with the following algorithm:
blaze query "allpaths($targets, $targets)"
to get a list of labels.blaze build --output_groups=haskell_dep_info
on those labels, and read the DepInfo
for each one. Ignore labels that don't have a DepInfo
, i.e., that aren't Haskell targets.DepInfo
(vertices are labels). This will give us a correct set which is refined from (1).Some things I'm not sure about;
deps($targets)
in step (1)?DepInfo
be better merged into CompileInfo
or LibraryInfo
?@aherrmann
shake
andghcide
are a different rule type,haskell_cabal_library
. That's probably why your query isn't picking them up. Can you try adding that (and alsohaskell_toolchain_library
, I suppose) to the filter list?
I dug a bit deeper. The issue is not due to the filter that I added in #12. The filter for haskell_binary|haskell_library|haskell_test
is only applied to getAllIntermediateTargets
. I.e. only affects packages that should be loaded by source. ghcide
and shake
are external dependencies and should be loaded as compiled packages. (haskell_cabal_library
also doesn't return haskell_compile_info
)
The compiled dependencies are determined by the query defined in buildDependentPackages
. In this case it looks as follows (as shown by --show-commands
)
bazel cquery 'let ts = (//compiler/damlc/daml-lf-conversion:daml-lf-conversion + (//compiler/damlc/stable-packages:generate-stable-package + (//compiler/damlc/daml-ide:daml-ide + (//compiler/damlc/daml-compiler:daml-compiler + (//compiler/damlc/daml-ide-core:daml-ide-core + (//compiler/damlc:damlc-lib + //compiler/damlc:damlc-bootstrap)))))) in (kind('\''haskell_library|haskell_proto_library|haskell_toolchain_library|haskell_cabal_library'\'', deps($ts, 1)) - $ts)'
Running this query indeed doesn't include ghcide
or shake
in the listed dependencies, even though both are direct dependencies of //compiler/damlc:damlc-lib
which is included in ts
.
It turns out that @stackage//:ghcide
and @stackage//:shake
are vendored_packages
in the corresponding stack_snapshot
call. I.e. they are alias
rules that point to haskell_cabal_library
somewhere else. But the query that hrepl
applies doesn't handle alias
.
Thanks for outlining the DepInfo
approach. That looks like it could work, though I haven't thought about the three open questions you raise in the bottom, yet. I think this might also provide a solution to the alias
issue I've described above.
hrepl
attempts to build thehaskell_compile_info
output group of non-Haskell targets even if they are not explicitly listed on the command-line.This issue can be reproduced on the
daml
repository as follows:Both targets that are specified on the command line are
haskell_binary
targets. The failing target//compiler/damlc/stable-packages:gen-stable-packages
is agenrule
target.Setting the
--show-commands
flag shows that that target first appears after the firstallpaths
query:Indeed,
//compiler/damlc:damlc-bootstrap
has adata
dependency on//compiler/damlc/stable-packages
which is afilegroup
capturing the outputs of thegenrule
//compiler/damlc/stable-packages:gen-stable-packages
which in turn has atools
dependency on//compiler/damlc/stable-packages:generate-stable-package
.One could argue that it doesn't make sense to try and load both those binaries into one GHCi session. Unfortunately, that doesn't prevent this issue as both
haskell_binary
targets share common library dependencies. E.g. the following will trigger the same issuewhere
//compiler/damlc/daml-lf-conversion
is ahaskell_library
target.Unfortunately, I'm not aware of a
cquery
command to filter out targets that don't provide a certain output group. However, one possible way to avoid this issue would be to filter for Haskell rules before attempting to build the output groups. E.g.kind("haskell_binary|haskell_library|haskell_test", ...)
forcompile_info
andkind("haskell_library|haskell_cabal_library", ...)
forlibrary_info
. This would potentially also allow users to pass wildcards tohrepl
, e.g.hrepl revision 33f879eecd986c92ba67f31aec05e3d7f4fbc025 daml revision 761efbe0856941ab398925846dee0d4bbf0b1811