More complete included records

opoudjis commented 10 months ago

As part of the updates to https://github.com/metanorma/metanorma-ietf/issues/95 , I am implementing IETF RFC rendering of references that include other references, such as IETF BCP 9.

For the rendering to be grammatical, two elements are mandatory for the included references:

A title
An author

Neither are currently present; e.g. for BCP 9:

<bibdata type="standard" schema-version="v1.2.8">
...
  <relation type="includes">
    <bibitem>
      <formattedref format="text/plain">RFC2026</formattedref>
      <docidentifier type="IETF" primary="true">RFC 2026</docidentifier>
    </bibitem>
  </relation>
...

Ideally, the complete record should be included; RFC XML expects it. Is it possible to include the title and publisher, at least, of the included bibitems?

opoudjis commented 10 months ago

I have put in a workaround of setting missing title as [TITLE] and author as empty, but the [TITLE] at least needs to be addressed. This is how BCP 9 is currently rendering:

   [a]        "[TITLE]", RFC 2026.
              "[TITLE]", RFC 5657.
              "[TITLE]", RFC 6410.
              "[TITLE]", RFC 7100.
              "[TITLE]", RFC 7127.
              "[TITLE]", RFC 7475.
              "[TITLE]", RFC 8789.
              "[TITLE]", RFC 9282.
              <https://www.rfc-editor.org/info/bcp9>

andrew2net commented 10 months ago

@opoudjis we have 3 IETF datasets: relaton-data-ids, relation-data-rfc, and relation-data-rfcsubseries. The IETF BCP 9 reference that you mentioned is in the relation-data-rfcsubseries. Do we need to include title and publisher into relation for all 3 datasets?

FYI only relaton-data-rfc has publishers. The relation-data-ids has only authors. The relation-data-rfcsubseries doesn't have contributors at all.

opoudjis commented 10 months ago

What xml2rfc ends up doing is breaking up the IETF BCP document into its constituent RFC references, printing them all in sequence, and printing "BCP 9" to the left of the first one, to indicate that all those RFC documents belong to the one BCP 9 document.

The multi-document standards are IETF BCP and IETF STD documents. That's a question for you, but I would assume that it's only relation-data-rfc documents (RFCs) being included in multi-document standards within relation-data-rfcsubseries (BCP, STD). Authors in the constituent RFC documents from relation-data-rfc are needed, and so is the RFC publisher. I don't think anything else is needed.

andrew2net commented 10 months ago

The relaton-data-rfcsubseries is updated:

$ relaton fetch --no-cache 'BCP 9'
[relaton-ietf] (BCP 9) Fetching from Relaton repository ...
[relaton-rss] Downloaded index from https://raw.githubusercontent.com/relaton/relaton-data-rfcsubseries/main/index-v1.zip
[relaton-ietf] (BCP 9) Found: `BCP 9`
<bibdata type="standard" schema-version="v1.2.8">
  <fetched>2024-01-15</fetched>
  <title format="text/plain" language="en" script="Latn">Best Current Practice 9</title>
  <uri type="src">https://www.rfc-editor.org/info/bcp9</uri>
  <docidentifier type="IETF" primary="true">BCP 9</docidentifier>
  <docnumber>BCP0009</docnumber>
  <language>en</language>
  <script>Latn</script>
  <relation type="includes">
    <bibitem type="standard">
      <title type="main" format="text/plain">The Internet Standards Process -- Revision 3</title>
      <uri type="src">https://www.rfc-editor.org/info/rfc2026</uri>
      <docidentifier type="IETF" primary="true">RFC 2026</docidentifier>
      <docidentifier type="DOI">10.17487/RFC2026</docidentifier>
      <docnumber>RFC2026</docnumber>
      <date type="published">
        <on>1996-10</on>
      </date>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">S. Bradner</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="publisher"/>
        <organization>
          <name>RFC Publisher</name>
        </organization>
      </contributor>
      <contributor>
        <role type="authorizer"/>
        <organization>
          <name>RFC Series</name>
        </organization>
      </contributor>
      <language>en</language>
      <script>Latn</script>
      <abstract format="text/html" language="en" script="Latn">
        <p>This memo documents the process used by the Internet community for the standardization of protocols and procedures. It defines the stages in the standardization process, the requirements for moving a document between stages and the types of documents used during this process. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements.</p>
      </abstract>
      <series>
        <title format="text/plain">BCP</title>
        <number>9</number>
      </series>
      <series>
        <title format="text/plain">RFC</title>
        <number>2026</number>
      </series>
      <series type="stream">
        <title format="text/plain">IETF</title>
      </series>
      <keyword>Protocols</keyword>
      <keyword>copyrights</keyword>
      <keyword>intellectual</keyword>
      <keyword>property</keyword>
    </bibitem>
  </relation>
  <relation type="includes">
    <bibitem type="standard">
      <title type="main" format="text/plain">Guidance on Interoperation and Implementation Reports for Advancement to Draft Standard</title>
      <uri type="src">https://www.rfc-editor.org/info/rfc5657</uri>
      <docidentifier type="IETF" primary="true">RFC 5657</docidentifier>
      <docidentifier type="DOI">10.17487/RFC5657</docidentifier>
      <docnumber>RFC5657</docnumber>
      <date type="published">
        <on>2009-09</on>
      </date>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">L. Dusseault</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">R. Sparks</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="publisher"/>
        <organization>
          <name>RFC Publisher</name>
        </organization>
      </contributor>
      <contributor>
        <role type="authorizer"/>
        <organization>
          <name>RFC Series</name>
        </organization>
      </contributor>
      <language>en</language>
      <script>Latn</script>
      <abstract format="text/html" language="en" script="Latn">
        <p>Advancing a protocol to Draft Standard requires documentation of the interoperation and implementation of the protocol. Historic reports have varied widely in form and level of content and there is little guidance available to new report preparers. This document updates the existing processes and provides more detail on what is appropriate in an interoperability and implementation report. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements.</p>
      </abstract>
      <relation type="updates">
        <bibitem>
          <formattedref format="text/plain">RFC2026</formattedref>
          <docidentifier type="IETF" primary="true">RFC2026</docidentifier>
        </bibitem>
      </relation>
      <series>
        <title format="text/plain">BCP</title>
        <number>9</number>
      </series>
      <series>
        <title format="text/plain">RFC</title>
        <number>5657</number>
      </series>
      <series type="stream">
        <title format="text/plain">IETF</title>
      </series>
      <keyword>rfc2026</keyword>
      <keyword>2026</keyword>
      <keyword>guidance</keyword>
      <keyword>interoperation</keyword>
      <keyword>implementation</keyword>
      <keyword>reports</keyword>
      <keyword>advancement</keyword>
      <keyword>draft standard</keyword>
    </bibitem>
  </relation>
  <relation type="includes">
    <bibitem type="standard">
      <title type="main" format="text/plain">Reducing the Standards Track to Two Maturity Levels</title>
      <uri type="src">https://www.rfc-editor.org/info/rfc6410</uri>
      <docidentifier type="IETF" primary="true">RFC 6410</docidentifier>
      <docidentifier type="DOI">10.17487/RFC6410</docidentifier>
      <docnumber>RFC6410</docnumber>
      <date type="published">
        <on>2011-10</on>
      </date>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">R. Housley</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">D. Crocker</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">E. Burger</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="publisher"/>
        <organization>
          <name>RFC Publisher</name>
        </organization>
      </contributor>
      <contributor>
        <role type="authorizer"/>
        <organization>
          <name>RFC Series</name>
        </organization>
      </contributor>
      <language>en</language>
      <script>Latn</script>
      <abstract format="text/html" language="en" script="Latn">
        <p>This document updates the Internet Engineering Task Force (IETF) Standards Process defined in RFC 2026. Primarily, it reduces the Standards Process from three Standards Track maturity levels to two. This memo documents an Internet Best Current Practice.</p>
      </abstract>
      <relation type="updates">
        <bibitem>
          <formattedref format="text/plain">RFC2026</formattedref>
          <docidentifier type="IETF" primary="true">RFC2026</docidentifier>
        </bibitem>
      </relation>
      <series>
        <title format="text/plain">BCP</title>
        <number>9</number>
      </series>
      <series>
        <title format="text/plain">RFC</title>
        <number>6410</number>
      </series>
      <series type="stream">
        <title format="text/plain">IETF</title>
      </series>
    </bibitem>
  </relation>
  <relation type="includes">
    <bibitem type="standard">
      <title type="main" format="text/plain">Retirement of the "Internet Official Protocol Standards" Summary Document</title>
      <uri type="src">https://www.rfc-editor.org/info/rfc7100</uri>
      <docidentifier type="IETF" primary="true">RFC 7100</docidentifier>
      <docidentifier type="DOI">10.17487/RFC7100</docidentifier>
      <docnumber>RFC7100</docnumber>
      <date type="published">
        <on>2013-12</on>
      </date>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">P. Resnick</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="publisher"/>
        <organization>
          <name>RFC Publisher</name>
        </organization>
      </contributor>
      <contributor>
        <role type="authorizer"/>
        <organization>
          <name>RFC Series</name>
        </organization>
      </contributor>
      <language>en</language>
      <script>Latn</script>
      <abstract format="text/html" language="en" script="Latn">
        <p>This document updates RFC 2026 to no longer use STD 1 as a summary of "Internet Official Protocol Standards". It obsoletes RFC 5000 and requests the IESG to move RFC 5000 (and therefore STD 1) to Historic status.</p>
      </abstract>
      <relation type="updates">
        <bibitem>
          <formattedref format="text/plain">RFC2026</formattedref>
          <docidentifier type="IETF" primary="true">RFC2026</docidentifier>
        </bibitem>
      </relation>
      <series>
        <title format="text/plain">BCP</title>
        <number>9</number>
      </series>
      <series>
        <title format="text/plain">RFC</title>
        <number>7100</number>
      </series>
      <series type="stream">
        <title format="text/plain">IETF</title>
      </series>
    </bibitem>
  </relation>
  <relation type="includes">
    <bibitem type="standard">
      <title type="main" format="text/plain">Characterization of Proposed Standards</title>
      <uri type="src">https://www.rfc-editor.org/info/rfc7127</uri>
      <docidentifier type="IETF" primary="true">RFC 7127</docidentifier>
      <docidentifier type="DOI">10.17487/RFC7127</docidentifier>
      <docnumber>RFC7127</docnumber>
      <date type="published">
        <on>2014-01</on>
      </date>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">O. Kolkman</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">S. Bradner</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">S. Turner</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="publisher"/>
        <organization>
          <name>RFC Publisher</name>
        </organization>
      </contributor>
      <contributor>
        <role type="authorizer"/>
        <organization>
          <name>RFC Series</name>
        </organization>
      </contributor>
      <language>en</language>
      <script>Latn</script>
      <abstract format="text/html" language="en" script="Latn">
        <p>RFC 2026 describes the review performed by the Internet Engineering Steering Group (IESG) on IETF Proposed Standard RFCs and characterizes the maturity level of those documents. This document updates RFC 2026 by providing a current and more accurate characterization of Proposed Standards.</p>
      </abstract>
      <relation type="updates">
        <bibitem>
          <formattedref format="text/plain">RFC2026</formattedref>
          <docidentifier type="IETF" primary="true">RFC2026</docidentifier>
        </bibitem>
      </relation>
      <series>
        <title format="text/plain">BCP</title>
        <number>9</number>
      </series>
      <series>
        <title format="text/plain">RFC</title>
        <number>7127</number>
      </series>
      <series type="stream">
        <title format="text/plain">IETF</title>
      </series>
      <keyword>Guidance</keyword>
      <keyword>Standards</keyword>
      <keyword>Standards Process</keyword>
      <keyword>Advancement</keyword>
      <keyword>Proposed Standard</keyword>
    </bibitem>
  </relation>
  <relation type="includes">
    <bibitem type="standard">
      <title type="main" format="text/plain">Increasing the Number of Area Directors in an IETF Area</title>
      <uri type="src">https://www.rfc-editor.org/info/rfc7475</uri>
      <docidentifier type="IETF" primary="true">RFC 7475</docidentifier>
      <docidentifier type="DOI">10.17487/RFC7475</docidentifier>
      <docnumber>RFC7475</docnumber>
      <date type="published">
        <on>2015-03</on>
      </date>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">S. Dawkins</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="publisher"/>
        <organization>
          <name>RFC Publisher</name>
        </organization>
      </contributor>
      <contributor>
        <role type="authorizer"/>
        <organization>
          <name>RFC Series</name>
        </organization>
      </contributor>
      <language>en</language>
      <script>Latn</script>
      <abstract format="text/html" language="en" script="Latn">
        <p>This document removes a limit on the number of Area Directors who manage an Area in the definition of "IETF Area". This document updates RFC 2026 (BCP 9) and RFC 2418 (BCP 25).</p>
      </abstract>
      <relation type="updates">
        <bibitem>
          <formattedref format="text/plain">RFC2026</formattedref>
          <docidentifier type="IETF" primary="true">RFC2026</docidentifier>
        </bibitem>
      </relation>
      <relation type="updates">
        <bibitem>
          <formattedref format="text/plain">RFC2418</formattedref>
          <docidentifier type="IETF" primary="true">RFC2418</docidentifier>
        </bibitem>
      </relation>
      <series>
        <title format="text/plain">BCP</title>
        <number>9</number>
      </series>
      <series>
        <title format="text/plain">RFC</title>
        <number>7475</number>
      </series>
      <series type="stream">
        <title format="text/plain">IETF</title>
      </series>
    </bibitem>
  </relation>
  <relation type="includes">
    <bibitem type="standard">
      <title type="main" format="text/plain">IETF Stream Documents Require IETF Rough Consensus</title>
      <uri type="src">https://www.rfc-editor.org/info/rfc8789</uri>
      <docidentifier type="IETF" primary="true">RFC 8789</docidentifier>
      <docidentifier type="DOI">10.17487/RFC8789</docidentifier>
      <docnumber>RFC8789</docnumber>
      <date type="published">
        <on>2020-06</on>
      </date>
      <contributor>
        <role type="editor"/>
        <person>
          <name>
            <completename language="en" script="Latn">J. Halpern</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="editor"/>
        <person>
          <name>
            <completename language="en" script="Latn">E. Rescorla</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="publisher"/>
        <organization>
          <name>RFC Publisher</name>
        </organization>
      </contributor>
      <contributor>
        <role type="authorizer"/>
        <organization>
          <name>RFC Series</name>
        </organization>
      </contributor>
      <language>en</language>
      <script>Latn</script>
      <abstract format="text/html" language="en" script="Latn">
        <p>This document requires that the IETF never publish any IETF Stream RFCs without IETF rough consensus. This updates RFC 2026.</p>
      </abstract>
      <relation type="updates">
        <bibitem>
          <formattedref format="text/plain">RFC2026</formattedref>
          <docidentifier type="IETF" primary="true">RFC2026</docidentifier>
        </bibitem>
      </relation>
      <series>
        <title format="text/plain">BCP</title>
        <number>9</number>
      </series>
      <series>
        <title format="text/plain">RFC</title>
        <number>8789</number>
      </series>
      <series type="stream">
        <title format="text/plain">IETF</title>
      </series>
      <keyword>process</keyword>
      <keyword>publication</keyword>
    </bibitem>
  </relation>
  <relation type="includes">
    <bibitem type="standard">
      <title type="main" format="text/plain">Responsibility Change for the RFC Series</title>
      <uri type="src">https://www.rfc-editor.org/info/rfc9282</uri>
      <docidentifier type="IETF" primary="true">RFC 9282</docidentifier>
      <docidentifier type="DOI">10.17487/RFC9282</docidentifier>
      <docnumber>RFC9282</docnumber>
      <date type="published">
        <on>2022-06</on>
      </date>
      <contributor>
        <role type="author"/>
        <person>
          <name>
            <completename language="en" script="Latn">B. Rosen</completename>
          </name>
        </person>
      </contributor>
      <contributor>
        <role type="publisher"/>
        <organization>
          <name>RFC Publisher</name>
        </organization>
      </contributor>
      <contributor>
        <role type="authorizer"/>
        <organization>
          <name>RFC Series</name>
        </organization>
      </contributor>
      <language>en</language>
      <script>Latn</script>
      <abstract format="text/html" language="en" script="Latn">
        <p>In RFC 9280, responsibility for the RFC Series moved to the RFC Series Working Group and the RFC Series Approval Board. It is no longer the responsibility of the RFC Editor, and the role of the IAB in the RFC Series is altered. Accordingly, in Section 2.1 of RFC 2026, the sentence "RFC publication is the direct responsibility of the RFC Editor, under the general direction of the IAB" is deleted.</p>
      </abstract>
      <relation type="updates">
        <bibitem>
          <formattedref format="text/plain">RFC2026</formattedref>
          <docidentifier type="IETF" primary="true">RFC2026</docidentifier>
        </bibitem>
      </relation>
      <series>
        <title format="text/plain">BCP</title>
        <number>9</number>
      </series>
      <series>
        <title format="text/plain">RFC</title>
        <number>9282</number>
      </series>
      <series type="stream">
        <title format="text/plain">IETF</title>
      </series>
      <keyword>Protocols</keyword>
      <keyword>copyrights</keyword>
      <keyword>intellectual</keyword>
      <keyword>property</keyword>
    </bibitem>
  </relation>
</bibdata>

ronaldtse commented 10 months ago

I have put in a workaround of setting missing title as [TITLE] and author as empty, but the [TITLE] at least needs to be addressed. This is how BCP 9 is currently rendering:
   [a]        "[TITLE]", RFC 2026.
              "[TITLE]", RFC 5657.
              "[TITLE]", RFC 6410.
              "[TITLE]", RFC 7100.
              "[TITLE]", RFC 7127.
              "[TITLE]", RFC 7475.
              "[TITLE]", RFC 8789.
              "[TITLE]", RFC 9282.
              <https://www.rfc-editor.org/info/bcp9>

For the record, BCPs do not currently have titles. They may be assigned titles in the future, according to @rjsparks.

When there is no title, "BCP 9" or "Best Current Practice 9" should be the title.

rjsparks commented 10 months ago

BCPs do not have titles or authors. BCPs are collections and they are mutable. They contain RFCs (that are not).

We have been very hand-wavy with this over time, and have a long history of showing the title and authors for the only RFC that's in a BCP, but the practice does not generalize to all BCPs.

We have had BCPs "disappear" in the sense that they went to containing zero rfcs.

We have process BCPs that we refer to that have dozens of constituent RFCs. Egg-shell-walking around the issue has let us continue to refer (badly IMO) to them by the title and authors of one of the original constituent RFCs.

This is a reality to deal with. Changing that reality will be an arduous journey through the process stakeholder's approval paths.

opoudjis commented 10 months ago

@ronaldtse and @rjsparks I'm afraid you've both misunderstood the concern I raised, which is an internal one.

BCPs do indeed have no title, and there is no provision for such a title in RFC XML referencegroup. The titles and IDs supplied are of the included documents. The problem I was having was that our bibliographic tool was not passing on to me the titles of the standards included in the BCP, and that was why I needed to put in "[TITLE]".

This issue has now been fixed with this ticket: the BCP 9 reference is being generated from Metanorma input:

[bibliography]
== Bibliography
* [[[a,IETF BCP 9]]]

as

   [a]        "The Internet Standards Process - Revision 3", BCP 9, RFC
              2026, IETF.

              "Guidance on Interoperation and Implementation Reports for
              Advancement to Draft Standard", BCP 9, RFC 5657, IETF.

              "Reducing the Standards Track to Two Maturity Levels",
              BCP 9, RFC 6410, IETF.

              "Retirement of the "Internet Official Protocol Standards"
              Summary Document", BCP 9, RFC 7100, IETF.

              "Characterization of Proposed Standards", BCP 9, RFC 7127,
              IETF.

              "Increasing the Number of Area Directors in an IETF Area",
              BCP 9, RFC 7475, IETF.

              "IETF Stream Documents Require IETF Rough Consensus",
              BCP 9, RFC 8789, IETF.

              "Responsibility Change for the RFC Series", BCP 9, RFC
              9282, IETF.

              <https://www.rfc-editor.org/info/bcp9>

The inserted "BCP 9" is happening because it is, after all, a document identifier, but it is redundant, and I'm reopening https://github.com/metanorma/metanorma-ietf/issues/95 to get rid of it.

rjsparks commented 10 months ago

Say more about specifically what you are planning to get rid of?

Please read the discussion and worked examples at https://github.com/ietf-tools/xml2rfc/issues/1067 and make sure you are going the same direction.

opoudjis commented 9 months ago

Say more about specifically what you are planning to get rid of?

The repeated "BCP 9" in the references enumerated under BCP 9, because it's redundant, and extraneous to the actual references; e.g. in

"The Internet Standards Process - Revision 3", BCP 9, RFC
              2026, IETF.

Please read the discussion and worked examples at ietf-tools/xml2rfc#1067 and make sure you are going the same direction.

aaaand from that ticket, I'm undoing that change. I see you're also expecting authors, DOIs, dates, and URLs; that sounds like we have to look at our population of BCP again.

relaton / relaton-ietf

More complete included records #120