Spack environment: cannot combine spec cflags with compiler cflags

[scheibelp]

Steps to reproduce

Using the following spack.yaml:

spack:
  # add package specs to the `specs` list
  specs:
  - zlib cflags=-O3
  view: true
  concretizer:
    unify: true
  'compilers:':
  - compiler:
      spec: gcc@=9.4.0
      paths:
        cc: /usr/bin/gcc
        cxx: /usr/bin/g++
        f77: null
        fc: null
      flags:
        cflags: "-What"
      operating_system: ubuntu20.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []

(in my case I did this on an Ubuntu 20 docker image)

and do

spack env create t1 spack.yaml 
spack -e t1 concretize -f -U

(nothing happens)

Error message

There is no error message, but nothing is concretized, and no spack.lock appears.

If I remove the flag from either the spec in the spec list, or set cflags to "" in the compiler entry, then the concretization succeeds.

If I run in debug mode (spack -d...) then most of the output looks like normal config loading, although the following unusual message appears (not reported as an error and doesn't show up without -d):

==> [2023-06-30-23:22:35.561111] UNKNOWN SYMBOL: attr("node_compiler", zlib, gcc)

Information on your system

$ spack debug report
* **Spack:** 0.21.0.dev0 (0c2b98ca7044588d56e9e4c9c178f411b6dc15c1)
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-skylake
* **Concretizer:** clingo

General information

• [X] I have run spack debug report and reported the version of Spack/Python/Platform
• [X] I have searched the issues of this repo and believe this is not a duplicate
• [X] I have run the failing commands in debug mode and reported the output

[scheibelp]

Note that combining the compiler config with command-line invocation works:

spack -e t1 spec zlib cflags=-O3
Concretized
--------------------------------
 -   zlib@1.2.13%gcc@9.4.0 cflags="-What -O3" +optimize+pic+shared build_system=makefile arch=linux-ubuntu20.04-skylake

but the environment will not concretize.

[adrienbernede]

@becker33 this is preventing me from moving to environments: I have specs with extra flags on top of those defined at compiler definition.

[adrienbernede]

@becker33 @scheibelp @tldahlgren Please allow me to insist on the fact that this is critical for RADIUSS open-source software at LLNL. We can’t move on to environments because of this. Thank you.

[scheibelp]

I can look into this next week

[alalazo]

Adding some more information for the record, as I'm looking into this while reviewing #42655.

The bug here is in FlagMap.satisfies (as @scheibelp notes in #41049) , specifically in:

Spec("zlib cflags='-What -O3'").satisfies("zlib cflags='-O3'")

which returns False, while a True is expected in:

https://github.com/spack/spack/blob/3d1d5f755ff0c0f39764620d3c769b41bd786c67/lib/spack/spack/solver/asp.py#L469

From there the spec builder takes the wrong branch and so we get an "unused spec" while we expected none.

[alalazo]

I tried to look into the docs if we define somewhere what should be the semantic of combining flags from different sources, but couldn't find anything. Did I miss any relevant section?

Basically, my thoughts are that for flags order is important and it seems we don't specify anywhere how:

1. Command line flags
2. Compiler flags
3. Flags from packages

are composed.

[scheibelp]

if we define somewhere what should be the semantic of combining flags from different sources, but couldn't find anything. Did I miss any relevant section?

I don't think we do. IMO it would be easier to bluntly append all these sources in a fixed order, at least to start, and raise an error if the combined list has conflicting flags. Overall the approach I suggest is:

• There can be an optional initial step to disqualify specs for reuse if they reorder flags
  • (only do this if they are reordered WRT a single source, presumably ordering between sources that aren't conflicts aren't of concern to the user)
• Solver/spec satisfaction is based on subsets (not ordering and ignoring conflicts like -O2/-O3) - which I think means it would work exactly like multivalued variants
  • The solver remembers each source though, and flags are ordered (a) by source and then (b) their order as was manually specified within the source
• Raise an error after solver completes if flag conflicts are detected
  • Add an option to ignore flags from compilers.yaml on a per-package basis
• If we get this far, I think the spec is consistent WRT what the user asked for (e.g. if we are reusing, then we're using something compatible)

I predict this will work smoothly so long as package.py files don't start encoding cflags into their depends_on statements (there are currently no cases of this in the builtin repo).

~The~ Some use cases to consider:

• Installed spec foo cflags="-O3 -werror", does it satisfy foo cflags="-O3" request?
  • What about installed foo cflags="-O3 -g"
• Installed spec foo ldflags="-la -lb", does it satisfy foo ldflags="-lb -la"?
• Make sure we throw an error if user asks for foo cflags="-w" and the only available compilers have cflags=-werror
  • These flags have fully-opposing effects (moreso than -O2 and -O3)
• User wants most packages using compiler %gcc to get -O2, but wants -O3 for a specific package
• Combining cflags (assuming no conflicts) from
  • require:
  • command line spec
    • including cflag propagation
  • depends_on
  • conflicts(cflags=...)?

[alalazo]

@scheibelp Do we define the meaning of satisfies, intersects and constrain for compiler flags in the approach you propose above? It's not clear to me that we do so, and I think it's important to have a clear model (if we have that all the rest will follow).

Taking into account as a simple example the optimization flag -O3, gcc returns the following:

$ gcc -Q -O3 --help=optimizers | grep ftree-pre
  -ftree-pre                        [enabled]
$ gcc -Q -fno-tree-pre -O3 --help=optimizers | grep ftree-pre
  -ftree-pre                        [disabled]

So does "-fno-tree-pre -O3".satisfies("-O3") ? In my opinion the correct answer should be "no".

The reasoning is that, if the answer is "yes" we can disable explicitly another optimization from -O3 until, by induction, we disabled them all. At that point "<no-optimizations>".satisfies("-O3"). That's clearly a wrong model.

In principle, I think we should try to get a canonical form for the flags[^1] and have the flags behave like a multi-valued variant. I'm not sure though this is easy to obtain for each compiler we support.

If we agree on a model, but that is difficult to implement, we can opt for some "wrong" (or approximate) implementation that is better for our "practical" purposes. But at least we'll know what we would like to obtain from the model, and how the implementation falls short of that.

[^1]: e.g. the one obtained with gcc -Q ... --help=<section> on GCC

[scheibelp]

If we agree on a model, but that is difficult to implement, we can opt for some "wrong" (or approximate) implementation that is better for our "practical" purposes. But at least we'll know what we would like to obtain from the model, and how the implementation falls short of that.

I think the easiest way to define the model is to describe the use cases we want to satisfy. I tried to do that in my proposal. I think more use cases will come up naturally as this proposal (or any proposal) draws criticism (i.e. I think defining an implementation will help us talk about how to model things).

In principle, I think we should try to get a canonical form for the flags and have the flags behave like a multi-valued variant.

I take "canonical" to mean a unique representation: do you mean that for a given unordered set of flags, that we should know how to order them? I don't think we can actually guarantee doing that (at least, we can't do it for -lx -ly for example because lib orderings can have different meanings).

Do we define the meaning of satisfies, intersects and constrain for compiler flags in the approach you propose above?

In my proposal .satisfies does not handle issues like the possible conflict between -fno-tree-pre and -O3: For two flag specifications X and Y, X.satisfies(Y) if all flags in Y appear in X.

does "-fno-tree=pre -O3".satisfies("-O3")? In my opinion the correct answer should be "no".

This example is interesting (do you want me to break out all use cases into a separate comment here?): this means that e.g. there is no conflict if -fno-tree-pree -O3 comes from one source, but it does conflict if it is assembled from two different sources, which makes it impossible to model w/ our logic for multi-valued variants (e.g. it is not covered by conflicts("cflags=-O2", when "cflags=-O3").

I think the answer should be "yes" in terms of what the solver and "spec.satisfies" says, and I think my proposal can make that work:

• If we want to prevent an installed spec like -fno-tree-pre -O3 from being used for -O3, we can omit it from the set of reusable specs before the solve (I say "can" because I think users might actually want to say that -fno-tree-pre -O3 satisfies -O3)
• If the command line spec is -fno-tree-pre -O3 and compilers.yaml is -O3, then (according to my proposal) the result is e.g. -fno-tree-pre -O3 -O3 and we can detect an error (point being, we have a notion of -O3 appearing w/different sources and can raise an error after the solver completes). If the command line is the only place to mention -O3, there is no conflict and we're good

The reasoning is that, if the answer is "yes" we can disable explicitly another optimization from -O3 until, by induction, we disabled them all.

I think my explanation for -fno-tree-pre addresses this, but let me know if you think it doesn't

[alalazo]

do you mean that for a given unordered set of flags, that we should know how to order them?

My idea is that from a given ordered set of flags, we can get a unique unordered set that represents it. gcc has ways to obtain that representation for different set of flags (e.g. optimizations, target specific flags, warnings), where it expands options like -OX into the single options that are enabled or disabled, and it reduces conflicting flags. Not sure if other compilers expose anything similar.

[scheibelp]

My idea is that from a given ordered set of flags, we can get a unique unordered set that represents it.

This might be true of optimization flags, but isn't true for linking flags. We could still do it for the optimization flags, but committing ourselves to this level of fidelity seems problematic:

• This canonicalization (even if it exists for all compilers) may change between compilers and between compiler versions
• Build systems can always undermine us (unless we go out of our way to strip all flags given to our compiler wrappers, they can insert whatever flags they like at build time)

so it feels like we'd be doing a lot of work for something that can immediately be undone.

Hopefully this does not come across as dismissive: I think what you point out is important, but I think that the best response to this specific issue is for us to explicitly agree not to handle it.

[alalazo]

but isn't true for linking flags.

Correct. Also, any path-like flag doesn't fit in this model. Are you aware of any use of ldflags="-L/path1 -L/path2 -lfoo -lbar" or similar? Not seeing them in here I thought we dealt with customizing flags like that in other ways. If we need to model linking flags too in specs, then I don't think we can map flags to multi-valued variants. At least not in a formal model (which may differ from our implementation).

I agree though that we can avoid handling certain cases and be aware of that, which means our model has deficiencies users need to be aware of. That wouldn't be dissimilar from the current way e.g. compilers are treated.

[adrienbernede]

@alalazo @scheibelp it looks like this is turning into deeper design considerations that the issue first suggested.

I think I get the issue about the complexity of comparing a list of flags. One thing I am not sure to understand however, is why would spack have 2 different specs to compare here in the first place.

@alalazo you said the issues is that Spec("zlib cflags='-What -O3'").satisfies("zlib cflags='-O3'") returns false. And the discussion then demonstrates that there is no obvious policy, since -What could actually alter the meaning of -O3.

My (possibly naive) question is as follow. Given the environment in this reproducer, to me it looks like there is only one spec to consider: zlib cflags='-What -O3'. Then why should we care about satisfying zlib cflags='-O3' ?

[alalazo]

Then why should we care about satisfying zlib cflags='-O3' ?

It's the abstract spec requested by the user. We check that to associate a concrete spec with the corresponding abstract spec.

[adrienbernede]

@alalazo I understand what you’re saying, but still, by asking for this compiler (defined with the additional flag -What) the user is asking for -O3 -What even if that is done implicitly.

To reuse your point above, if we had 2 compilers, one with no flags and one with -fno-tree-pre, by choosing the second one, the user is expecting "slightly less than -O3".

I am not saying I am right to think that way, but to me, it looks like you are debating defining "satisfies" in a way that will either suppose full knowledge of how compilers deal with flags, or accept compromise, while I think the solution might be to have the compilers flags applied to the abstract spec because that’s what the users asked (knowingly or not).

[alalazo]

@adrienbernede I am fine with a workaround to get past this problem (I think @scheibelp is working on one, happy to review it when ready), but the above considerations mean to me we don't have a clean formal model of flags. This makes discussing flags more difficult, because flags are inherently ordered, and later flags could potentially alter the meaning or previous flags.

I could go with counterexamples to your "slightly less than -O3" but I guess you got the point: it's difficult to define "slightly less", or make sure "slightly less" means the same thing for all the users, without a formal model :slightly_smiling_face:

[adrienbernede]

@alalazo my comments may just be too naive. I'll trust you with doing what's best for spack.

[alalazo]

my comments may just be too naive. I'll trust you with doing what's best for spack.

I wouldn't dismiss them as too naive. I'm just trying to explain why I'm looking for a model for flags which defines how Flag.satisfies and Flag.constrain work. Currently a flag satisfies another if it is exactly the same string which, as this issue proofs, is too restrictive and prevents composing flags from different sources.

That said, one practical solution might be to explicitly agree to not handle the semantic inconsistencies discussed above, as @scheibelp says proposes in https://github.com/spack/spack/issues/38664#issuecomment-1967866664 We may get a model which is not 100% satisfactory, but we'll be documenting how the model works and in which cases it may give unexpected results. This would still be a big improvement from the state in this issue.

[scheibelp]

I am fine with a workaround to get past this problem (I think @scheibelp is working on one, happy to review it when ready),

To be clear: I don't have a workaround yet. My only strategy so far in this thread was to decide how to handle concretization of flags. Exploring workarounds is also worthwhile IMO:

• My first thought is to provide a config option that allows us to use a loose definition of satisfaction for flags (i.e. when that config option is active, use an alternative definition of FlagMap.satisfies from https://github.com/spack/spack/pull/41049): the solver already appears to combine flags from all sources and produces a consistent order for them (see asp...reorder_flags - I haven't 100% confirmed this is complete for all combinations of sources)

• Part of my suggested solution above was to allow different flag sources to optionally override one another. That was intended to be more helpful in combination with a mechanism that is generally willing to combine flag groups that don't conflict as part of concretization, but could be taken separately. The hierarchy would be

  • command line specs.
  • requirements from packages.yaml
  • flags in compilers.yaml

  Overall this approach is challenging because these sources are not fully distinguished in concretize.lp.

[scheibelp]

Compiler flag combination semantics

The following attempts to describe cases where two sets of compiler flags are complementary based on my reading of GCC. Loosely speaking, something like

-O3 -Wall and -O3

are complementary. In words, I would take complementary to mean cases where one list of compiler flags would be accepted in place of another.

@alalazo this is my attempt to more-fully capture the flag semantics so that we may be aware of the possible deficiencies in an implementation. Overall, I still think https://github.com/spack/spack/issues/38664#issuecomment-1963541947 is a fully-specified and reasonable implementation for dealing with flag combination.

Modifiers: one flag partially undoing the effect of another

• Optimization flags can "interfere" with each other:
  • -O3 is modified by -fno-tree-pre
  • More loosely, one might say -g also modifies -O3
• Likewise, warning/error flags can interfere with each other (e.g. -w -Wabi)
• Users may or may not want to combine flags from different sources that modify one another
  • i.e. if they ask for -O3, they may be fine with (or in fact want) -O3 -fno-tree-pre

Conflicts: flags that are precisely opposite in their effect

• Flags can directly conflict but this is rare
  • e.g. -ftree-pre and -fno-tree-pre
  • "group" flags where a separate combination of -no-x effectively removes all items in the group
    • e.g. -O3 -fno-tree-pre ...
    • e.g. -Werror and -Wno-error=x ...
    • e.g. -w -Wall?
  • "level" based flags:
    • -O2 and -O3
    • -gctf0 and -gctf2
    • ? can we detect this by looking for "\d+" at the end of options?
      • as in, do all options with numbers at the end represent levels
  • For modifiers that are not optimization flags, one order is always wrong
    • -Werror -Wno-error=x makes sense, but -Wno-error=x has no effect if you put it first
  • Multiple instances of an option with different values
    • e.g. -std=c11 and -std=gnu11
• Preprocessor macros might conflict but this cant be detected
  • e.g. related to enabling support for different GPU libraries (the writers of the code may have assumed only one such macro would be defined)
• Most other flags don't technically conflict
  • -w (don't generate warnings) and -Werror (all warnings are errors)
• Users would never want direct conflicts

Ordering: when does it matter

• Order matters for
  • Linking flags (-lfoo, --start-group)
  • Search flags (-I, -L)
  • Direct conflicts
  • All modifiers except for optimization flags
    • GCC has specific logic for handling optimization flags in any order
    • For something like -Werror -x -y -z (where each of x/y/z are of the form -Wno-error=...) note that ordering amongst x/y/z doesn't matter, but rather whether they come before or after -Werror (as pointed out in the conflicts section above)
• Overall, this means we may want to allow the user to provide input on inter-source ordering (e.g. flags from compilers.yaml vs. flags from packages.yaml vs. flags from depends_on)

The --start-group linking flag

(UPDATE March 21: I added this section)

--start-group -la -lb --end-group --start-group -lc -ld --end-group is different from --start-group -la -lb -lc -ld --end-group

This would complicate a set-based representation because --start-group appears multiple times (and that is important for the linking behavior).

However, for purposes of representation in the concretizer, one could consolidate into a pseudo flag like --ld-group="-la -lb" --ld-group="-lc -ld".

I made a separate section for this because I don't know any other flags that have this behavior.

[scheibelp]

Broad approach based on discussion of above between @tgamblin @alalazo @psakievich @becker33 (each step here can be a separate PR):

• first: relax .satisfies to be subset
• next phase: refactor/unify multivalued-variant and flag logic
  • We discussed that step 1 makes this very similar to multi-valued variants with at least 2 exceptions: (a) propagation (compiler flags don't propagate across compilers) (b) compiler flags have to remember where they came from and be applied together w/ all flags from their source (e.g. if you say require: -O2 -Wall, when=... if you get -O2 from this source you also need -Wall). Anyhow the point is there's likely some refactoring/consolidation we can do to simplify
• next phase: add knobs for override semantics
  • If compilers.yaml applies -O3 and we want -O2 for a specific package, we want to be able to add a per-package config option to disable flags from compilers.yaml, or possibly to tell Spack that flags from the package replace compilers.yaml flags where there are conflicts. The main point is that our semantics start out simple: we don't attempt to prevent -O2 and -O3 from both appearing and we don't automatically alter the set of flags; we anticipate that we can provide simple options that give users more fine-grained control where these conflicts occur.
• next phase: reuse - there are potential complications w/ reuse
  • we want to start out by saying a reuse spec is applicable if it supplies at least all of the flags requested (so again, subset)
  • there are potential side issues though (e.g. if a reuse spec has "-O3 -O2" it doesn't technically satisfy a request for "-O3"
  • It is anticipated we can mitigate this with a pre-filtering step like https://github.com/spack/spack/pull/42782

[koysean]

Since I don't see a workaround mentioned in this thread, here's one that I've found useful in the meantime.

From the original issue, this fails:

  specs:
    - zlib cflags="-O3"

but using a require: statement to set the needed cflags does concretize and the concretized spec concatenates the flags from the compiler and package configurations:

  packages:
    zlib:
      require: 'cflags="-O3"'
  specs:
    - zlib

This doesn't take care of any flag conflicts or ordering. It seems it also can't propagate flags to dependencies automatically like you could with an abstract spec (with cflags==...). However, it's been good enough for what I need to do.