Missing predefined text in Normative References (sub-clause in Section)

[Intelligent2013]

Source issue: https://github.com/metanorma/metanorma-iso/issues/1091#issuecomment-2120820053

The source document (ISO_7005_2_1988.zip) contains Normative references as sub-clause in the Section General:

[type=section]
== General

[heading=scope]
=== Scope

This part of ISO 7005 for a single of flanges specifies requirements
for circular grey, malleable and ductile cast iron
flanges in the following nominal pressure ratings: ...

[bibliography,heading=normative references]
=== Normative references

* [[[iso185,ISO 185:1988]]], _Classification of grey cast iron_

Issue 1: The presentation XML doesn't contain the default predefined text in the Normative references sub-clause:

            <references id="_normative_references" normative="true" obligation="informative">
                <title depth="2">1.2<tab/>Normative references</title>
                <bibitem id="iso185" type="standard" schema-version="v1.2.8">
                    <formattedref>
                        <em>
                            <span class="stddocTitle">Grey cast iron — Classification</span>
                        </em>
                    </formattedref>

If I add the user-defined boilerplate text:

[bibliography,heading=normative references]
=== Normative references

[.boilerplate]
--
The following standards contain provisions which, through
reference in this text, constitute provisions of this International
Standard. At the time of publication, the editions indicated
were valid. All standards are subject to revision, and parties to
agreements based on this International Standard are encouraged
to investigate the possibility of applying the most recent
editions of the standards listed below. Members of IEC and ISO
maintain registers of currently valid International Standards.
--

* [[[iso185,ISO 185:1988]]], _Classification of grey cast iron_

then

Issue 2: the presentation XML contains the user-defined boilerplate text in the element note(type="boilerplate"):

            <references id="_normative_references" normative="true" obligation="informative">
                <title depth="2">1.2<tab/>Normative references</title>
                <note id="_d11262e7-8568-d279-93d7-b6d5cd60c5a5" type="boilerplate">
                    <name>NOTE</name>
                    <p id="_6826c404-d40a-afe2-4fa0-6cef224e3af8">The following standards contain provisions which, through
reference in this text, constitute provisions of this International
Standard. At the time of publication, the editions indicated
were valid. All standards are subject to revision, and parties to
agreements based on this International Standard are encouraged
to investigate the possibility of applying the most recent
editions of the standards listed below. Members of IEC and ISO
maintain registers of currently valid International Standards.</p>
                </note>
                <bibitem id="iso185" type="standard" schema-version="v1.2.8">
                    <formattedref>
                        <em>
                            <span class="stddocTitle">Grey cast iron — Classification</span>
                        </em>
                    </formattedref>

instead of just p.

If I remove the top-level Section:

[type=section]
== General

(and, sure, remove = before each sub-clause title), then the boilerplate text generates correctly in the both cases.

[opoudjis]

Sadly, ISO historically did different and daft things, and for reasons that I do not comprehend or approve of, we are graverobbing old ISO standards.

It looks like the Metanorma code dealing with boilerplate for reference clauses is expecting them to be top-level clauses, and I can see why it would. Investigating.

[opoudjis]

Yeah, this is... bad.

The code was structured from the beginning with the presupposition that bibliographies would be banished to back material, under //bibliography/references, even if the normative references were to appear as clause 2. When the bibliography is embedded in a subclause like this, Metanorma still recognises it as a bibliography, but it is no longer located where it expects to find it, in back matter. So I need to allow Metanorma to find bibliographies anywhere in the document. We already are dealing with that in NIST, which puts bibliographies in annexes.

[opoudjis]

This is disrupting my existing code structure, and it is forcing code redesign. Which means you are not going to get it immediately, @Intelligent2013

The structure at the moment of the elements is:

• references can only contain bibitem: they are the leaf node of references sections
• If bibliographies are to contain nested clauses, the nesting clauses are clause: /bibliography/clause/clause/.../references
• So if we are to insert the boilerplate anywhere, it will be at the topmost clause containing a references

But if bibliographies can be shoved into an arbitrary subclause, we can't do that: in the example above, "General" is not a bibliographic clause.

Instead, we need a rationalised way of inferring boilerplate location, and ways of overriding it, as we pretty much already have with Terms and definitions.

• Boilerplate goes to the highest clause that contains references or clause[@type = 'boilerplate'] and no other clause or terms.
• We need to support, consistently for terms and references, not just boilerplate override text and blank boilerplate, but also "insert the default boilerplate text here".
• Be consistent with whether boilerplate is a note or a paragraph or a clause. I thought I had ended up making it consistently a note for inserted boilerplate (as you are seeing, @Intelligent2013), and I don't see why that should be a problem. Is it?

[opoudjis]

Per https://github.com/metanorma/metanorma-plateau/issues/32, which also does this clause embedding, for terms:

I am therefore retracting the ISO inference that the closest ancestor clause of terms is where the boilerplate goes. If authors want to get creative with clauses containing both terms and non-term clauses, they are going to have to signal themselves where they want the boilerplate to go explicitly, with an "insert the default boilerplate text here" directive.

[opoudjis]

So: universally, in Metanorma,

• Terms boilerplate by default goes to the highest clause that contains terms or clause[@type = 'boilerplate'] or and no other clause type (including definitions)
• References boilerplate by default goes to the highest clause that contains references or clause[@type = 'boilerplate'] and no other clause type
• The preferred manner of introducing boilerplate is note[@type = 'boilerplate'], and has been since metanorma-standoc 2.3.0
• An empty note[@type = 'boilerplate'] or clause[@type = 'boilerplate'] communicates that no boilerplate text will be supplied
• An note[@type = 'boilerplate'] or clause[@type = 'boilerplate'] containing the text (default) communicates that the boilerplate text supplied by default by the flavour needs to appear here. Any normal logic applied to the boilerplate, e.g. modifying it based on the presence of external terms, is applied in generating that text.

[opoudjis]

Done for standoc, will now need to adjust iso code to integrate the changes.

[opoudjis]

There's an idiosyncratic rule in standoc: "places single terms boilerplate at root if there are clauses preceding the terms collection, other than boilerplate". So

      == Terms and definitions

      [.nonterm]
      === Terms0
      Boilerplate

      === Terms

      ==== term

      === Symbols

<sections>
<clause>
<-- BOILERPLATE
<clause/>
<terms/>
</clause>
</sections>

That rule is not going to withstand what is happening with nested terms within other clauses, and needs to be dispensed with. We will need a new override rule: if we know a terms clause to be a terms clause because of its heading, we will retain that information, and still make it the default destination of boilerplate, so long as it contains clauses and terms, but not if it also contains symbols. That can now be overridden, but we will have:

== Terms and definitions

<-- Boilerplate

=== Clause

=== Term 1

== Terms, definitions and Symbols

=== Clause

=== Terms

<-- Boilerplate

==== Term 1

=== Symbols

So the default algorithm now becomes:

• Start with the known terms section
• Go upwards so long as all children of the clause are terms
• Keep going upwards if an ancestor is a terms container (was titled "terms and..." something), and the siblings of terms are clauses
• Stop going upwards if a child is Symbols

That means:

== General

=== Terms

<-- boilerplate

but

== General

=== Terms

<-- boilerplate

=== Preamble

==== Normal Terms

==== Other Terms

This is horrific, but we do need a default that can cope with Plateau and old ISO, and the existing default won't. And at least now we can override the default behaviour.

[opoudjis]

To do this, assign type=terms on clauses that are recognised as terms containers

[opoudjis]

And we do not rename to Terms and definitions clauses that are not recognised as terms containers (i.e. heading=Terms and definitions)

[opoudjis]

Issue 1 now resolved:

    it "places boilerplate in Normative References subclause" do
      input = <<~INPUT
        #{ASCIIDOC_BLANK_HDR}

        [type=section]
        == General

        [heading=scope]
        === Scope

        This part of ISO 7005 for a single of flanges specifies requirements
        for circular grey, malleable and ductile cast iron
        flanges in the following nominal pressure ratings: ...

        [bibliography,heading=normative references]
        === Normative references
      INPUT
      output = <<~OUTPUT
        #{BLANK_HDR}
        <sections>
            <clause id="_" type="section" inline-header="false" obligation="normative">
              <title>General</title>
              <clause id="_" type="scope" inline-header="false" obligation="normative">
                <title>Scope</title>
                <p id="_">This part of ISO 7005 for a single of flanges specifies requirements
        for circular grey, malleable and ductile cast iron
        flanges in the following nominal pressure ratings: …<200b></p>
              </clause>
              <references id="_" normative="true" obligation="informative">
                <title>Normative references</title>
                <p id="_">There are no normative references in this document.</p>
              </references>
            </clause>
          </sections>
        </iso-standard>
      OUTPUT
      expect(xmlpp(strip_guid(Asciidoctor.convert(input, *OPTIONS))))
        .to be_equivalent_to xmlpp(output)
    end

[opoudjis]

Issue 2 now also resolved:

    it "places user-defined boilerplate in Normative References subclause" do
      input = <<~INPUT
        #{ASCIIDOC_BLANK_HDR}

        [type=section]
        == General

        [heading=scope]
        === Scope

        This part of ISO 7005 for a single of flanges

        [bibliography,heading=normative references]
        === Normative references

        [.boilerplate]
        --
        The following standards contain provisions
        --
      INPUT
      output = <<~OUTPUT
        #{BLANK_HDR}
        <sections>
            <clause id="_" type="section" inline-header="false" obligation="normative">
              <title>General</title>
              <clause id="_" type="scope" inline-header="false" obligation="normative">
                <title>Scope</title>
                <p id="_">This part of ISO 7005 for a single of flanges</p>
              </clause>
              <references id="_" normative="true" obligation="informative">
                <title>Normative references</title>
                <p id="_">The following standards contain provisions</p>
              </references>
            </clause>
          </sections>
        </iso-standard>
      OUTPUT
      expect(xmlpp(strip_guid(Asciidoctor.convert(input, *OPTIONS))))
        .to be_equivalent_to xmlpp(output)
    end