[intro.races] p13 `shall` vs. `is`

[xmh0511]

Full name of submitter (unless configured in github; will be published with the issue): Jim X

[intro.races] p13 says:

The value of a non-atomic scalar object or bit-field M, as determined by evaluation B, shall be the value stored by the visible side effect A.

The note says

[Note 12: If there is ambiguity about which side effect to a non-atomic object or bit-field is visible, then the behavior is either unspecified or undefined. — end note]

shall means if the rule is violated, the implementations shall diagnose per [intro.compliance.general] p2, which means the program is ill-formed. Both unspecified and undefined behavior does not violate the rule. Consider the case where there would be no visible side effect to the value computation B, which would happen to cause the program data race per [intro.races] p21, which is defined as UB, however, this situation has first violated the shall, which means the implementation should first issue one diagnostic message, which means we never fall into UB.

Suggested Resolution

we may want to say

The value of a non-atomic scalar object or bit-field M, as determined by evaluation B, is the value stored by the visible side effect A.

[frederick-vs-ja]

I think other occurences in [intro.races] are also problematic. Perhaps we can resolve this editorially.

[languagelawyer]

shall means if the rule is violated, the implementations shall diagnose per [intro.compliance.general] p2, which means the program is ill-formed.

This is very wrong.

Lets start from the end of the quote. A program is ill-formed when it is not well-formed. A well-formed program is «C++ program constructed according to the syntax and semantic rules». I've highlighted constructed. [intro.races] p13 has nothing to do with «construction» of a program.

Speaking of «shall diagnose», [intro.compliance.general]/(2.3) says «if a program contains a violation of any diagnosable rule». [intro.races] p13 is not about «program containing» anything.

And the last, but the most important thing. About «shall». [intro.scope]/1

This document specifies requirements for implementations of the C++ programming language.

The ISO C++ Standard is drafted in accordance with the editorial rules of the ISO/IEC Directives, Part 2.

7 Verbal forms for expressions of provisions 7.1 General The user of the document shall be able to identify the requirements he/she is obliged to satisfy in order to claim conformance to a document. The user shall also be able to distinguish these requirements from other types of provision (recommendations, permissions, possibilities and capabilities). ... 7.2 Requirement The verbal forms shown in Table 3 shall be used to express requirements. Table 3: Preferred verbal form: «shall»/«shall not»

So, «shall» in

The value of a non-atomic scalar object or bit-field M, as determined by evaluation B, shall be the value stored by the visible side effect A.

indicates that this what an implementation is required to satisfy to claim the conformance. I.e. this shall be read as

[In/On a confirming implementation (executing the program)] The value of a non-atomic scalar object or bit-field M, as determined by evaluation B, shall be the value stored by the visible side effect A.

This is not something which can be «violated» (by a user of the implementation) so that it could be «diagnosed» (by the implementation), it is a requirement which determines the conformance of an implementation.

@jensmaurer could you please stop any attempts to remove «shall» here and in similar cases? This would be disaster.

[jensmaurer]

The problem we have for the C++ standard is that it defines requirements on two levels: Requirements on the implementation, and requirements on (well-formed) programs. This is a rather unique situation for programming languages, which is not well-reflected in the (rather general) ISO Directives.

We admit as much in [intro.compliance.general] p2:

Although this document states only requirements on C++ implementations, those requirements are often easier to understand if they are phrased as requirements on programs, parts of programs, or execution of programs.

Our operation mode for the past decade (at least) has been to clarify requirements on implementations by saying "an implementation shall ..." where the emphasis is desirable, and otherwise just describe the behavior of C++ using "is" etc. For example, [expr.add] p3 says "The result of the binary + operator is the sum of the operands." even though it is a requirement on the implementation that the + operator "shall" yield a certain result.

We use "shall" to describe diagnosable rules for programs. For example, [expr.add] p1 says

For addition, either both operands shall have arithmetic type, or one operand shall be a pointer to a completely-defined object type and the other shall have integral type.

If we can't use "shall" for that purpose, we'd have to use "the program is ill-formed if ..." phrasings everywhere, which (depending on context) might just be longer and more verbose or actually awkward to phrase and harder to understand.

Obviously, the rules for atomic operations behavior are requirements on the implementation (not diagnosable rules for programs), so "shall" is undesirable per the above.

[languagelawyer]

We admit as much in [intro.compliance.general] p2:

Although this document states only requirements on C++ implementations, those requirements are often easier to understand if they are phrased as requirements on programs, parts of programs, or execution of programs.

I know. Often, not always.

The value of a non-atomic scalar object or bit-field M, as determined by evaluation B, shall be the value stored by the visible side effect A.

clearly reads as a requirement put directly on an implementation. So shall here is perfectly fine. Even required, according to the ISO rules.

We use "shall" to describe diagnosable rules for programs.

diagnosable ≠ shall be diagnosed, since a program ≠ an execution of a program. http://eel.is/c++draft/intro.compliance.general#2.3 speaks about the former, not the later.

[jensmaurer]

since a program ≠ an execution of a program

Exactly, and that's why we shy away from "shall" for runtime behavior (there's no way to give a diagnostic on the violation of runtime things, and we have had confusion in that area in the past).

[languagelawyer]

and that's why we shy away from "shall" for runtime behavior (there's no way to give a diagnostic on the violation of runtime things

There is no requirement to diagnose "shall" violation, which is not "contained" in a program, but "happened'" during program execution. Per http://eel.is/c++draft/intro.compliance.general#2.3, diagnostics is required only for "contained" violations.

But I can see the problem, yes. There is nothing in http://eel.is/c++draft/intro.compliance.general#2 saying what an implementation shall do when a semantic rule is violated at runtime.

P.S. Just in case. "shall for runtime behavior" is "shall" used when the standard describes requirement for an implementation in terms of requirements on a(n execution of a) program. I don't know good examples, I'll link https://timsong-cpp.github.io/cppwp/n4868/dcl.ref#5.sentence-3 (CWG453). The paragraph which is the subject of this non-issue is not an example of "shall for runtime behavior", the "shall" here is used to describe the requirements for an implementation directly. So, there is no problem in this paragraph.

[jensmaurer]

CWG453 is exactly a case where "shall" is used and it looks like a compile-time-diagnosable situation, but it actually isn't (because "valid object" isn't known until runtime). And thus, "shall" should not be used. The proposed drafting replaces it with "undefined behavior", which clearly says "the user wrote a bad program".

There is no requirement on the implementation involved.

[tkoeppe]

(Clarifying how and when we use "shall", and how the specification works in general, is something we can do in a new "Introduction" clause.)

[languagelawyer]

CWG453 is exactly a case where "shall" is used and it looks like a compile-time-diagnosable situation

IDK why necessary as "a compile-time-diagnosable situation"

We have

Although this document states only requirements on C++ implementations, those requirements are often easier to understand if they are phrased as requirements on programs, parts of programs, or execution of programs

so this could be read as requirement on an implementation executing the program. Except that what we want to say is that there is no requirement. And this is because the use "shall" there is wrong. Not because someone may thing that "diagnostics" is involved every time they see "shall" (without NDR).

But after all, I think we agree that "shall" in CWG453 and "shall" here are very different cases, the former one is bad and the later one is good. Better and more correct than "is".

[languagelawyer]

Clarifying how and when we use "shall"

If you use it as specified in ISO/IEC Directives, Part 2, no need, since it is referenced in the Foreword.

If you gonna use it incompatibly with ISO/IEC Directives, Part 2, then

This document was drafted in accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).

will need to be removed from Foreword. (But I suspect then ISO won't approve "This document" as an ISO standard?)

[RealLitb]

I can see where @languagelawyer is coming from. He seems to argue that the program does not "contain" a violation of "A reference shall be initialized to refer to a valid object or function" when it contains int &ref = *(int*)nullptr; because as @jensmaurer says, objects only exist at runtime and @languagelawyer quoted a text that distinguishes between those concepts: "those requirements are often easier to understand if they are phrased as requirements on programs, parts of programs, or execution of programs".

I did never actually distinguish between these, but can see his point.

[languagelawyer]

@RealLitb note I'm not saying that the use of "shall" in CWG453 is a good thing, it would be extremely hard to reformulate it as a positive requirement on an implementation [when executing a program]. So, instead, the lack of requirement should be indicated. I can't even come with a good example of using "shall" to express requirements in "execution of programs" spelling. I think, "shall" is not used in this spelling because of this, not because "it is impossible to diagnose". Why worry when it is not even required to diagnose?

In @jensmaurer example "The result of the binary + operator is shall be the sum of the operands." would be perfectly fine (even more correct), this is not an example of imposing requirement in "execution of programs" spelling.

[jensmaurer]

But after all, I think we agree that "shall" in CWG453 and "shall" here are very different cases, the former one is bad and the later one is good.

We don't agree. I think the change requested in this issue is good (although probably less pressing to change than other occurrences of "shall"), and the following edit would not be an improvement:

The result of the binary + operator is shall be the sum of the operands.

Just to be extra-clear on the meta-level here:

I believe our long-standing practice is to reserve "shall" for diagnosable C++ language rules (= syntax requirements on user programs). We try to get rid of "shall" used in any other capacity, except when saying "the implementation shall". The word "implementation" must appear, not something that can be divined to be related to the implementation.

If you want to change that long-standing practice, this ticket (and the CWG issue process) is not the right scope for such a change. Feel free to write a paper suggesting an alternative with specific examples, submit it to WG21, and be prepared to join a meeting to discuss your paper.

[languagelawyer]

Feel free to write a paper suggesting an alternative with specific examples, submit it to WG21, and be prepared to join a meeting to discuss your paper.

I think someone who will want to remove «This document was drafted in accordance with the editorial rules of the ISO/IEC Directives, Part 2» from the Foreword will need to prepare a paper.

I do not want.

[frederick-vs-ja]

I think someone who will want to remove «This document was drafted in accordance with the editorial rules of the ISO/IEC Directives, Part 2» from the Foreword will need to prepare a paper.

I don't think anybody wants to do this. In any case, the intent is just clarifying how and when we impose requirements without "shall". The remain uses of "shall" should still be "drafted in accordance with the editorial rules of the ISO/IEC Directives, Part 2".

[languagelawyer]

how and when we impose requirements without "shall"

Please read again what I've cited above:

The verbal forms shown in Table 3 shall be used to express requirements. Table 3: Preferred verbal form: «shall»/«shall not»

I think even when expressing requirements on an implementation using other forms of spelling (as requirements on programs, executions of programs, etc.), "shall" still shall be used, so that reader could distinguish requirements from non-requirements.

[frederick-vs-ja]

The verbal forms shown in Table 3 shall be used to express requirements. Table 3: Preferred verbal form: «shall»/«shall not»

I think even when expressing requirements on an implementation using other forms of spelling (as requirements on programs, executions of programs, etc.), "shall" still shall be used, so that reader could distinguish requirements from non-requirements.

I think the cited content means that when «shall»/«shall not» are used, they shall be used to express requirements, but not the other way around - it doesn't mean that when requirements are expressed, «shall»/«shall not» shall be used.

[languagelawyer]

@frederick-vs-ja a more complete citation:

The user of the document shall be able to identify the requirements he/she is obliged to satisfy in order to claim conformance to a document. The user shall also be able to distinguish these requirements from other types of provision (recommendations, permissions, possibilities and capabilities).

It is essential to follow rules for the use of verbal forms so that a clear distinction can be made between requirements, recommendations, permissions, possibilities and capabilities. To avoid risk of misinterpretation, verbal forms that are not defined in Table 3 to Table 7 shall not be used for the expression of provisions.

So it is in both directions. (BTW, I don't understand how did you manage to read as you did; I may understand it is kinda possible to do it the other way around)

(Waiting for someone to claim that

verbal forms that are not defined in Table 3 to Table 7 shall not be used for the expression of provisions

does not prohibit the use of non-verbal forms to express requirements)

[frederick-vs-ja]

So it is in both directions. (BTW, I don't understand how did you manage to read as you did; I may understand it is kinda possible to do it the other way around)

One reason is that ISO/IEC Directives, Part 2 itself sometimes expresses requirements without "shall". E.g., in the following paragraph:

24.3 Numbering and subdivision Within a given clause or subclause, notes shall be numbered sequentially. The numbering restarts at each new subdivision. A single note in a subdivision shall not be numbered.

the plain "restarts" also expresses a requirement.

[jwakely]

FWIW the C standard has a different explanation for the double duty that "shall" serves:

In this International Standard, “shall” is to be interpreted as a requirement on an implementation or on a program; conversely, “shall not” is to be interpreted as a prohibition.

If a “shall” or “shall not” requirement that appears outside of a constraint or runtime-constraint is violated, the behavior is undefined. Undefined behavior is otherwise indicated in this International Standard by the words “undefined behavior” or by the omission of any explicit definition of behavior. There is no difference in emphasis among these three; they all describe “behavior that is undefined”.

But that doesn't necessarily help, because it doesn't tell you whether a given use of "shall" is a requirement on an implementation or on a program. And sometimes it's not clear whether it's either, e.g.

Each instance of a backslash character () immediately followed by a new-line character is deleted, splicing physical source lines to form logical source lines. Only the last backslash on any physical source line shall be eligible for being part of such a splice. A source file that is not empty shall end in a new-line character, which shall not be immediately preceded by a backslash character before any such splicing takes place.

The first "shall" doesn't seem to be a requirement on either the implementation or the program, and should be "is eligible" IMHO. That sentence also appears in C++. The next two uses are requirements on a program, and I think they're diagnosable rather than UB because it's a constraint (C17 3.8) ... but I'm not sure. It's far from clear IMHO.

In general, unless we're going to change every "shall" to "the implementation shall ensure" or "the program shall ensure", I think replacing uses of "shall" with clear statements of fact (as in https://github.com/cplusplus/draft/pull/6457) is an improvement. I certainly don't think it would be disastrous as claimed above. That's just silly hyperbole.

If interpreting the meaning of "shall" requires semantic nuances about whether something is "contained" in the program or "happens" during execution, or requires scrutinising whether the requirement applies to the implementation or the program, that will make the standard harder to read, not easier. I strongly agree with Jens that "The result of the binary + operator is shall be the sum of the operands." would not be an improvement.

We can just say how the language works, the binary + operator performs summation, the value of an object is the last value stored to it, and then [intro.compliance.general] says that an implementation shall correctly execute that (which is a "shall" in the ISO requirement sense, which an implementation must meet to claim conformance). We don't need to individually require that an implementation correctly implements every little piece of the language like summation and storing values to objects. We can state what those things mean in C++ and require that an implementation does all those things correctly.

[languagelawyer]

One reason is that ISO/IEC Directives, Part 2 itself sometimes expresses requirements without "shall".

1. ISO/IEC Directives, Part 2 might not be drafted according to ISO/IEC Directives, Part 2 (unlikely)
2. This can be an issue (likely)

So, you are gonna to use one unintentional mistake as an excuse to do many intentional? Is it some kind of revenge?

[jensmaurer]

ISO/IEC Directives, Part 2 might not be drafted according to ISO/IEC Directives, Part 2

This is exactly the case. See the foreword, page ix:

It is not the intention that the ISO/IEC Directives, Part 2 follow the same structure and drafting rules as International Standards and associated publications (referred to collectively as “documents”, see 3.1.1).