Support ITU-T Meeting "Contribution" document type

[ronaldtse]

Similar to https://github.com/metanorma/metanorma-itu/issues/375

[ronaldtse]

@taddhar could you please help provide a sample Contribution document for templating purpose? Thanks.

[taddhar]

Hi all, sure thing.

[taddhar]

Here is the base template

ITU-T Contribution - Base Template.docx

[taddhar]

The most important are the fields in the header and in particular one must register a number before submitting

So the Cnnnn because say C614

[taddhar]

Now there are different types of Contributions.

There are the general ones, someone wants to make a simple text about something. Here is an example: T22-SG17-C-0223!!MSW-E (2).docx

[taddhar]

Then a very common type of Contribution is a contribution for a new baseline revised text.

Here is an example: T22-SG17-C-0660!!MSW-E (1).docx

[taddhar]

And one that is one of the most complex one is a contribution for a New Work Item: T17-SG17-C-0593!!MSW-E.docx

[taddhar]

This one is a New Work Item for a Recommendation and note that the table in the middle is refering to a so-called A.1 template which is here Form-A.01-Annex-A-New-Rec-Justif-template (1).docx

[taddhar]

When a contribution is for a Technical Paper, a Technical Report or for a non normative text one needs to use an A.13 template which is here: Form-A.13-Annex-A-New_non-normative_Justif-template (1).docx

[taddhar]

Here is an example Contribution for a Technical Paper or Report T22-SG17-C-0639!!MSW-E.docx

[taddhar]

Please let me know for anything

[taddhar]

Hi @opoudjis I answered in #262 I believe

[opoudjis]

@taddhar As I am sure you will appreciate, Metanorma relies on rigorous and parsimonious structure. What you have provided is not enough to work from, and I require a more rigorous presentation of possible contribution types. I am trying to work out what the variation is between the different artefacts you have provided, and I should not be having to analyse a quite possibly limited number of documents to work out what is going on. That is your responsibility to provide.

I am particularly unhappy that you have not provided a definition of "baseline revised text", and with zero Google hits for +ITU +"baseline revised text", I really am being forced to guess what this means. You are familiar with ITU processes. I am not. It is not proper use of my time for me to have to spend hours Googling and guessing.

metanorma-itu has implemented ITU-T Editing Guidelines, which applies to ITU-T Recommendations. That is how contributions are going to be styled by default. If different styling applies, you need to tell me.

As far as I can work out, contributions can follow the following structure:

• Generic contributions (e.g. SG17-C-0223)
• Contributions involving target documents
  • Contributions that constitute revisions of existing published documents (??)
  • Contributions that constitute revisions of existing draft documents (which is what I am guessing "baseline revised text" means)
  • These reference an existing document identifier, and document type; e.g. SG17-C-660 is a revision of X.arch-design, a recommendation, in draft.
  • Contributions that constitute proposals for new documents
  • These reference a document type, and look different depending on the document type of the proposed document. The fact that the templates do not even capture this information, but rely on human-readable indications of it, shows that they are insufficiently rationalised
  • Recommendations include the A.1 template as an initial annex (e.g. SG17-C-0593)
  • Other, non-normative types include the A.13 template as an initial annex (e.g. SG17-C-0639)
  • Contributions involving target documents attach the target document as an annex

The target document shall be entered as a separate document, which shall be embedded into the contribution document as an annex; e.g.

SG17-C-0639

[appendix]
== A.13 justification for proposed Draft....

[appendix]
== Initial proposed Baseline text

embed::itu-t-tr.cs-sc.adoc[]

(Embedding is a Metanorma-specific means of including a document which preserves that document's metadata, and keep it a separate document.)

A revised draft document is NOT presented as an annex; we can realise that inconsistency, but it is an inconsistency with New Work Item proposals

SG17-C-0660

== Proposal
Broadcom would like Q1 to discussion this revision, get feedback and produce a new revision text for X.arch-design

<<<

embed::x.arch-design[]

The doctype shall be consistently contribution. The subdoctype shall be generic, published-document (??), baseline-revised-text, new-work-item. We will add a target-doctype, which indicates the type of target document, target-docidentifier, which is the identifier of the target document, where applicable, and target-doctitle.

So:

:doctype: contribution
:subdoctype: general
:docnumber: SG17-C-0223

:doctype: contribution
:subdoctype: baseline-revised-text
:docnumber: SG17-C-660
:target-doctype: recommendation
:target-docidentifier: X.arch-design

:doctype: contribution
:subdoctype: new-work-item
:docnumber: SG17-C-0593
:target-doctype: recommendation
:target-docidentifier: X.tod

:doctype: contribution
:subdoctype: new-work-item
:docnumber: SG17-C-0639
:target-doctype: technical-report
:target-docidentifier: TR.cs-ra
:target-doctitle: Collection of Security Concern to support X.cs-ra Cyber Security Reference Architecture

Forms A1 and A13 are pretty much the same table, with some different introductory guidance on populating them; these are going to be realised as the same table, with some Liquid templating to change what gets rendered, and the introductory comment are going to form part of a Metanorma template.

So

mn-template-itu, contribution.adoc will contain the following Liquid template, activated only in case of a new Work Item:

{%if subdoctype == "New Work Item" %}

[appendix]
{% if target-doctype == "Recommendation" %}
== A.1 justification for proposed Draft new {{ target-doctype }} {{ target-docidentifier }} 
{% else %}
== A.13 justification for proposed Draft new {{ target-doctype }} {{ target-docidentifier }} "{{ target-doctitle}}"
{% endif %}

|===
h| Question |  /* enter Question id */ h| Proposed new ITU-T {{ target-doctype }} | /* enter date */  
...

{% endif %}

@ronaldtse I'm sure would like to see the entire A.1/A.13 table automatically populated from metadata. I don't think that is a good use of programming at all, especially if differences creep in.

Once the target documents are decoupled from the contribution as embedded documents, and the A1/A13 forms are streamlined and partly populated in Liquid, the contribution becomes reasonably tractable.

[opoudjis]

@ronaldtse Would be useful for us to have the documents that @taddhar supplied converted sample documents for us to test contributions. But I don't need them immediately, and I will be experimenting with how best to structure these anyway. Templates are going to do a lot of the heavy lifting here for forms...

[taddhar]

@opoudjis I read with interest your elaborated comments above and I completely understand your perspective. Sadly I only published #520 now which I wanted to do earlier as I realized quickly by myself that this was not correct and I am not the right person for what you need. I can only see what I need from my relatively small perspective and will work to bring here the right people with the right authority so that you get the right input and do not need to do undue search or guessing. This is indeed neither your job, nor a good use of your time, and this is simply not the right way.

Of course, I know by experience that even if you have the right people in this repo, there will be a grey area on a number of things so my own 'pragmatism' might turn to not be too bad at the end but let's make this properly.

[taddhar]

So first YES I absolutely recognize and appreciate that

Metanorma relies on rigorous and parsimonious structure

this is a strong benefit that I completely understand and want this principle to stay bold.

[taddhar]

Ahh yes, the 'baseline revised text' ... what is it in a non rigourous description.

So say you have a contribution proposing to establish a new work item say X.bla with an initial proposed baseline text for a Recommendation and therefore an A.1 justification table. This initial baseline text must follow the skeleton template for an ITU-T Recommendation (I guess i forgot to send this one right? or you have it already as you are already able to produce Recommendations ?)

This contribution is produced and submitted for a certain meeting of a certain study group, say SG17. The contribution is published say as C650 before a certain cut date (12 days before SG17 meeting starts). And unless exceptions, once the cut date was passed C650 cannot be revised! If it is revised AFTER the cut date it can be considered as a 'late contribution' and in principle should be deferred to the NEXT SG17 meeting.

Then the meeting starts and a certain Question of SG17, say Q15, reviews the contribution C650 and agrees to establish the new work item X.bla. At that point during the meeting the Rapporteur of the Question publishes a TD, say TD2100 which contains:

• all the body of C650 without its contribution header including the justification, the gap analysis, the A.1 and the initial baseline text
• all the comments and modifications in revision mark
• and usually a preamble before that modified body as editors notes summarising all the comments received during the meeting

Then Q15 sends TD2100 to its upper Working Party that discusses again this TD2100 in Working Party Closing Plenary and if they agree to establish it it is sent to the SG17 closing plenary (the last day).

At the end of the SG17 meeting, in closing plenary, SG17 say agrees then to establish the new work item X.bla as part of the work program of Q15. At this moment X.bla is part of the work program and TD2100 is used as its initial 'text' including an initial baseline text which is the one in the Annex of TD2100 which means the annex of C650 modified by Q15 in TD2100.

Then X.bla will appear in the work program. I imagine you can access to the current work program of Q15 but not to the content of the files because you don't have membership access I imagine: https://www.itu.int/ITU-T/workprog/wp_search.aspx?isn_sp=8265&isn_sg=8275&isn_qu=8396&isn_status=-1,8,1,3,7&details=0&field=acdefghijo

For example you may be able to see the one I produced as X.icd-schemas using metanorma and liquid templates thanks to all your help! https://www.itu.int/ITU-T/workprog/wp_item.aspx?isn=17986

Now at the next SG17 meeting someone wants to propose to improve the current baseline text which is in TD2100.

Typically you can do this with a new Contribution at the next meeting. In which case typically what I would do is create a new contribution using contribution template, then I would load the initial baseline text or the current baseline text, I would approve all the modifications, would put now in new revision mark mode and one could see all the new changes proposed.

Then I submit a Contribution say C850 which is NOT for a new work item but simply a contribution to revise the baseline text of an existing work item, in our case X.bla ... In fact others could do other contributions in parallel and in this case you end up in the meeting with 1 or multiple contributions to revise the new baseline text. When Q15 will open these contributions, the EDITORS of the work item, need to coordinate all the contributions and agree what is the final modifications. Then they produce a new TD, say TD3500 and then we agree at SG17 closing plenary that this is the new baseline text for this work item

etc. etc.

And this continues until it is decided that the text is stable and then we go for the next step of the process: consent if AAP process, determination if TAP process. AAP = Accelerated Approval Process and TAP = Traditional Approval Process

Does it help you 'visualising' the baseline text improvements?

[taddhar]

Ahh in fact there is an authoritative source. Am not sure if you are aware of the A.1 Recommendation:

https://www.itu.int/rec/T-REC-A.1-201909-I/en

This may answer a LOT of your questions, including on groups, sub groups, etc.

[taddhar]

Ahh yes:

metanorma-itu has implemented ITU-T Editing Guidelines, which applies to ITU-T Recommendations. That is how contributions are going to be styled by default. If different styling applies, you need to tell me."

Yes I would believe that there are different styles for documents that are not Recommendations. First of all you have other published documents example:

• Technical Paper
• Technical Report
• User guide
• Implementation guide
• Roadmap document

If you take a step back the way I see it is that you could group all the documents in 3 first categories:

• Input documents: Contributions (which later transform as TDs), Liaison Statements incoming (which are in fact TDs)
• Process documents: Temporary Documents (TDs), Reports, Agenda, Minutes, etc. (all of them are TDs)
• Output documents: Recommendations, Technical Papers, etc. (as per above)

[taddhar]

Yes then I understand your approach on how you try to make a categorisation, great job and i see the need to 'translate a bit':

• Generic contributions (e.g. SG17-C-0223)

• Contributions involving target documents

  • Contributions that constitute revisions of existing published documents (??)

Ahh yes we didn't review that case ... Hmm, there are here some specialisation of New Work Items for example to do:

• A Revision of an existing published text (Recommendation, Technical Paper, etc.)
• An Amendment to an existing published text (Recommendation, Technical Paper, etc.)
• A Corrigendum of an existing published text (Recommendation, Technical Paper, etc.)
• A supplemente to an existing published text (Recommendation, Technical Paper, etc.)

• Contributions that constitute revisions of existing draft documents (which is what I am guessing "baseline revised text" means)

Yes this is the whole point of my early long coment

* These reference an existing document identifier, and document type; e.g. SG17-C-660 is a revision of X.arch-design, a recommendation, in draft.

Yes and it has a TD number ... I hope you can at least see the metadata in the example of the work program of Q15 as per abobe comment

• Contributions that constitute proposals for new documents

Yes Contributions that constitue proposals as NEW WORK ITEM for a specific target document (a Recommendation, a Technical Paper, etc. INCLUDING a Revision, an Amemdment, a Corrigendum, etc. ... yes I forgot that case actually)

* These reference a document type, and look different depending on the document type of the proposed document. The fact that the templates do not even capture this information, but rely on human-readable indications of it, shows that they are insufficiently rationalised

Yes I agree on your remark therefore my own complaint that we do not even have the right templates in word ... so each time one needs to take a Contribution template, then add justification, gap analysis, etc. then pick either A.1 or A.13, then add an initial baseline text ... how annoying.

* Recommendations include the A.1 template as an initial annex (e.g. SG17-C-0593)
* Other, non-normative types include the A.13 template as an initial annex (e.g. SG17-C-0639)

• Contributions involving target documents attach the target document as an annex

Am not sure what you mean here

[taddhar]

The target document shall be entered as a separate document, which shall be embedded into the contribution document as an annex; e.g.

Well if you only use Word (which we do now in ITU) we cannot do that BUT HERE this would be a GREAT benefit of Metanorma: "the lego brick approach" I would love it.

SG17-C-0639

[appendix]
== A.13 justification for proposed Draft....

[appendix]
== Initial proposed Baseline text

embed::itu-t-tr.cs-sc.adoc[]

(Embedding is a Metanorma-specific means of including a document which preserves that document's metadata, and keep it a separate document.)

Ok I am not too sure what it means yet because my knowledge of metanorma is too limited but I believe I understand and like your intention.

A revised draft document is NOT presented as an annex; we can realise that inconsistency, but it is an inconsistency with New Work Item proposals

SG17-C-0660

== Proposal
Broadcom would like Q1 to discussion this revision, get feedback and produce a new revision text for X.arch-design

<<<

embed::x.arch-design[]

The doctype shall be consistently contribution. The subdoctype shall be generic, published-document (??), baseline-revised-text, new-work-item. We will add a target-doctype, which indicates the type of target document, target-docidentifier, which is the identifier of the target document, where applicable, and target-doctitle.

So:

:doctype: contribution
:subdoctype: general
:docnumber: SG17-C-0223

:doctype: contribution
:subdoctype: baseline-revised-text
:docnumber: SG17-C-660
:target-doctype: recommendation
:target-docidentifier: X.arch-design

:doctype: contribution
:subdoctype: new-work-item
:docnumber: SG17-C-0593
:target-doctype: recommendation
:target-docidentifier: X.tod

:doctype: contribution
:subdoctype: new-work-item
:docnumber: SG17-C-0639
:target-doctype: technical-report
:target-docidentifier: TR.cs-ra
:target-doctitle: Collection of Security Concern to support X.cs-ra Cyber Security Reference Architecture

Forms A1 and A13 are pretty much the same table, with some different introductory guidance on populating them; these are going to be realised as the same table, with some Liquid templating to change what gets rendered, and the introductory comment are going to form part of a Metanorma template.

So

mn-template-itu, contribution.adoc will contain the following Liquid template, activated only in case of a new Work Item:

{%if subdoctype == "New Work Item" %}

[appendix]
{% if target-doctype == "Recommendation" %}
== A.1 justification for proposed Draft new {{ target-doctype }} {{ target-docidentifier }} 
{% else %}
== A.13 justification for proposed Draft new {{ target-doctype }} {{ target-docidentifier }} "{{ target-doctitle}}"
{% endif %}

|===
h| Question |  /* enter Question id */ h| Proposed new ITU-T {{ target-doctype }} | /* enter date */  
...

{% endif %}

@ronaldtse I'm sure would like to see the entire A.1/A.13 table automatically populated from metadata. I don't think that is a good use of programming at all, especially if differences creep in.

Once the target documents are decoupled from the contribution as embedded documents, and the A1/A13 forms are streamlined and partly populated in Liquid, the contribution becomes reasonably tractable.

@opoudjis I believe I follow your thinking and I understand the problem you raise for Contribution for new baseline text on an existing work item and my proposal could be to do some tries and see what is the best possible implementation.

In fact this is the part that today requires that one takes the initial document, makes modification and at the end you must do a WORD comparison, this is what I had to do for X.icd-schemas as I had sooooo many changes, it led to a very big Word file to open in revision mark ... and until I realized I could use simple rev marks, with full rev marks, my Mac CPU was litterally BURNING. But this was due to the specific of this work item.

[taddhar]

finally I am in Frankfurt on my way to Australia for IETF 119 so I do my best now that I am insulated from family, colleagues, etc. :-) I hope this helps a little bit @opoudjis

[opoudjis]

That helps, thank you @taddhar, although I will confess to still being a little hazy on how this will all come together. What I am going to try to put in motion, after my analysis and your explanation, is indeed going to be somewhat Lego-brick-like:

• To metanorma as a document formatting system, Input Documents, Process Documents and Output Documents are indistinguishable: they are all, as far as it's concerned, just Recommendations, or whatever other document type you are processing. (Which is what you want! You don't want different processing for different stages of what is underlyingly the same thing.)
• The distinction between Input Documents, Process Documents and Output Documents is rather made in how the document content is wrapped and presented to the end user. Output documents get the characteristic cover page, and are treated as standalone documents. Input Documents and Process Documents are inclusions, embedded inside a Contribution document. So the Recommendation draft is still parsed and rendered like a standalone document, but it doesn't get a cover page, and it is embedded inside of the contribution document.
• That's the point of the embed:[] directive, which we created to embed ISO and CEN standards within BSI standards localising those standards. BSI just adds a BSI coverpage and a national preface to those ISO/CEN standards. Likewise, the contribution document adds a coverpage, a table, contributor remarks, and Forms A.1/A.13 to the Recommendation text. But that Recommendation text is still a self-standing document being wrapped inside of another, and you can straightforwardly publish it autonomously by just compiling it autonomously.
• The A.1/A.13 forms will of course not be copy-pasted from an external source under Metanorma; they will be part of the template for contribution documents, and to the extent possible, they will be auto-populated out of document attributes.

I'm going to start with the General contributions, since they don't involve A.1/A.13 or embedded documents, but I know what to do for the first iteration of this work. The devil is always in the detail with this kind of work, and the Word output won't necessarily look identical to the samples; but it is still a more tractable workflow than what is in place now...

[opoudjis]

The things to be entered into the base template of ITU-T contributions are as follows; we are actually already capturing all of them, since they are expected for technical reports.

• Study Group. See https://github.com/metanorma/metanorma-itu/issues/467 : In general we should enter the study group by its ID and name, :group-acronym: SG17 and :group: Study Group 17. If the acronym/id is not supplied, of course, we can readily infer it given the number: Study Group nnn => SG 17.
  • The template supplied presupposes that all documents are SG 17. We are not going to be doing that.
• Document identifier, SG17-C*** . In this case, we can infer it : SG 17-Cn, where n is the :docnumber: and C is inherited from doctype contribution
• The study period is also presupposed in the template, but should be generated from :group-year-start: and :group-year-end:
• Language: :language:
• Place: :meeting-place:
• Date: :meeting-date:
• Question: :question:. In https://github.com/metanorma/metanorma-itu/issues/467, I said questions should be given in full, e.g. Q10/17: Identity management and telebiometrics architecture and mechanisms, "Q11/17: Generic technologies (such as Directory, PKI, formal languages, object identifiers) to support secure applications". We can of course extract the ids from the full question attribute, and even strip the initial Q. I'd rather we keep the Q in in the general metadata though, rather than insert it later. The fact that ITU identifiers questions as both Q1/17 and 1/17 is of course laxity that Metanorma is not going to encourage.
• Source: :source:
• Title: :doctitle:
• Contact name: :fullname:, :fullname_2:, :fullname_3:
• Contact organisation::affiliation:, :affiliation_2:, :affiliation_3:
• Contact country: :address:, :address_2:, :address_3:
• Contact telephone: :phone:, :phone_2:, :phone_3:
• Contact email: :email:, :email_2:, :email_3:

Contributions don't have subgroups and workgroups.

The abstract is marked up as normal, [abstract]

The remaining text in the template will go to the Metanorma template.

[opoudjis]

We already have :intended-type: in technical reports, with values R, C, TD. Ideally we would use that target document type as described above, but I don't see how we can extend R and D and TD as document categories for technical reports to recommendation-amendment and implementers-guide. In fact, I don't know from #146 what R and D and TD are at all. So we will stick to docsubtype to specify the target document type, and leave it out or use general if there is no attached document.

[opoudjis]

I'm internationalising the template, because we internationalise everything in ITU. That means that someone is going to need to proofread the internationalisations I'm doing in general.

[opoudjis]

We need to set the study group period with a default value if it is not supplied. That will be the current year, or the previous year if this is an odd numbered year, up to the year two years later. If we are in 2024, 2024-2026; if we are in 2025, 2024-2026.

[opoudjis]

Almost there:

Need to remove the page breaks (because Contributions do not have separate cover pages.)

[taddhar]

Woww, that is a lot of considerations @opoudjis ! Ok I will need time to understand it all

[opoudjis]

I've generated the generic contribution, without embedded document, for SG17-C-0223. I attach it, rendered in English and (for jollies) Russian. You will note that the initial table is entirely populated from document attributes, and that those document attributes were already defined for technical reports, which have the same structure.

I have other tasks to tend to, so I will come back to embedded documents next week; but I'm happy with where this got to.

This is not 100% replication of the source document format, but frankly, that's the effort this format of document merits, as an initial draft, and it's already taken up much of the week.

The PDF and HTML are going to be less precise, but I don't think they warrant the additional effort at this stage.

contrib-223-E.zip

compress-223-R.zip

[opoudjis]

We've done Output documents to date, per your taxonomy. I'm now implementing:

• Input documents
  • Generic contributions (done)
  • Contributions embedding draft output documents, as revisions
  • Contributions embedding draft output documents, as new work proposals (adding form A.1/A.13)

That at least is a document structure taxonomy that I can work with.

I sort of understand the process of proposals and TDs and revisions and whatnot that you laid out, but you are asking me to make sense of a very convoluted process, that only tangentially seems to affect what I will end up doing with document formatting; so I'm disinclined to tackle TDs. These sound like they will be edits of Word documents as Input documents, whatever else happens in Metanorma, and the amount of detail tracking of edits that TDs require sounds to me intractable and prolix. Appendix I of ITU A-1, the Rapporteur progress report format, is very discursive, and I'm not seeing how it can even be put into a template, let alone be machine-populated. At any rate, I have no idea what they even look like, and whether they structurally are any different from edited input documents at all (just adding more text to the input document preface). If they are not structurally different, then editing input documents really will be enough.

Next I am going to do a baseline revised text contribution. The text of the contribution is going to be C-0660, but the embedded document is not going to be X.arch, because I don't have Asciidoctor source for that, and I don't need it for a sample document; I am grabbing T-REC-P.1203-201710-I as a brief recommendation from mn-samples-itu.

We'll touch base on this when you're in town, but if things go well, I'll be well on the way to realising what is in this ticket by the time you are.

[opoudjis]

Just opened 660...

... Metanorma does not realise Word Track Changes, and will not: that is a HUGE undertaking to realise, and one that may well force us to reimplement Metanorma Word output from scratch. As you can understand, I am unwilling to entertain that possibility.

If you want Word Track Changes in anything that Metanorma generates as a baseline revised text contribution, you will need to:

• Get the Word document that Metanorma generates
• Get the original document
• Do a Compare Documents in Word
• Paste the output with Track Changes into the contribution document, replacing the Metanorma-generated output

If you're going to do that, of course, it calls into question generating revised text contribution through Metanorma at all. The embed mechanism doesn't really make sense, if that's what you expect: you might as well generate the before and after as two separate output documents, and then paste the comparison into the Contribution generated without embedding. So Metanorma only makes sense for the automatically generated preface template, in that case.

Of course, this is not an issue for New Work Items: the embedding approach there makes all the sense in the world. But if we are undertaking to support a very Microsoft Word-bound process, you need to do analysis on what that will end up looking like. It won't look identical to the existing Microsoft Word-bound process, because Metanorma is not Word...

Anyway, proceeding with 660, but there will be no Track Changes in what we generate, and I've explained why.

[opoudjis]

We're going to have trouble with the fact that New Work Items treat the embedded document as an appendix. Our embedding model to date has relied on the fact that any embedding content is prefatory to the embedded document, which is why the embedded document's numbering is preserved intact: it is the main document.

Putting Appendixes will instantly destroy that, and in fact shows the bankruptcy of treating the embedding + embedded document as a unitary document: the embedded document is an Appendix, but its clauses are numbered as if it is a self-standing document (and of course, it is: its content is merely cut and pasted into the embedding document, clause numbers and all).

That's going to be a major change to the handling of embedded documents, and it will likely force structural changes to Metanorma.

[opoudjis]

We're even getting that change forced in the case of baseline revised text contributions. Embedding does not generate a title page for the embedded document (title, keywords, summary), because embedding expects to attach prefatory content to a document, and render it as a unitary document. This embedding expects a title page on the embedded document; and embedded documents usually will.

Likewise, there is no table of contents for the embedded document: Metanorma expects that it is getting a unitary document, and that tables of contents are for that entire document. The table of contents here is specific to the embedded document, since it is after all a cut and paste of the embedded document into the embedding template.

This leads to an unpleasant situation: the Semantic XML of such documents is going to end up containing self-standing documents, which need to be processed separately. In fact, these are not embeddings at all: they are document collections, because document collections are how we keep documents self-standing in Metanorma. And we have never attempted to create a single Word document out of a document collection until now.

Embedding almost gives us the right output, as the attachment shows. It is "only" missing the title and table of contents of the embedded document. But I no longer think it will give us the full right answer, especially once appendixes start turning up. Making that title and table of contents turn up is harder than it looks, and it would be a kludge. I don't think I should be tweaking embedding to enable separate processing of embedded documents, or contaminating the Metanorma document model with embedding, when that's what document collections were designed for. So I am going to start looking into what it will take to get a document collection working instead.

contrib-660.doc.zip

[opoudjis]

We have had a similar instance to this in JIS commentaries, which are included in the same Word document as a standard but paginated differently, and with a different header.

But we need to formalise this in Metanorma for Word documents specifically, because this is going to keep coming up, and we need a coherent approach rather than ad hoc solutions (such as we arguably did for JIS).

In Metanorma, we have a concept of a document: something that follows the Metanorma document schema, with a preface, clauses, and annexes in that order. The clauses are numbered consecutively from beginning to end. There is a single cover page, a single instance of introductory boilerplate, and a single table of contents.

We have instances where what were originally separate documents are combined into a single document, and are processed as a single document. That is embedded documents. So whatever the numbering of the clauses in the source document, the resulting artefact is a single document, with a single numbering of clauses from beginning to end, and a single cover and intro page. The way they have been implemented to date, the embedded document is the main set of clauses, and any wrapping content is additional prefatory content (or potentially additional annexes). That's how there can still be a single numbering of clauses: there is a single document, with stuff added before and after it.

We also have had instances where multiple documents reference each other, and are compiled as a document collection, dynamically updating those cross-references as hyperlinks. These are treated as a document collection: they end up all being aggregated into a single XML artefact, and the cross-references are preprocessed across the entire collection, so that they render consistently as hyperlinks. But each individual document in the collection is processed as a standalone document: the only thing that links the documents together is the hyperlinks and an overall cover page and index.

Document collections to date have been implemented as HTML and PDF, not Word.

We have implemented the following to date:

• ITU Recommendation Annexes. To date, we have only ever processed these as standalone documents. We have never attempted to render them alongside their original document, and if we had, I think we would have run into this issue a lot earlier.
• ISO 10303. The various parts of this standard, e.g. ISO 10303-41, are standalone documents, published separately in HTML and PDF, with their own coverpage and clause numbering. But documents in this standard hyperlink each other, and are published as a web site. So they are realised as a document collection.
• BIPM SI Brochure. This contains two separate documents with the same content, in English and in French. Each document has its own title and intro page, and its own clause numbering. There is one instance in which they hyperlink each other. They too are realised as a document collection.
• BSI national prefaces, wrapping CEN and ISO documents. They are additional national prefaces and a BSI coverpage, prefixed to a CEN or ISO document; the CEN document may itself be a regional prefix attached to an ISO document. The result is a single document, with a single (BSI) cover page, and a single (ISO) clause numbering. This is what we created embedded documents for, and they are indeed primarily intended for national adaptions of international/regional standards—which do not alter the main body of the standard, nor do they present the international standard as a distinct document.
• JIS Commentaries. These have a distinct Word header, and distinct pagination, but in itself that is not enough to treat them as distinct documents: they are numbered as annexes to the main document. They could have been treated as embedded documents, but because they are annexes, and aren't circulated as discrete documents, I have not yet bothered to do even that much.

That brings us to ITU contributions. On the face of it, they are Embedded documents. They add prefatory material to an separately circulated output document, presenting it as a single document. They maintain the same document header, and consecutive page numbering.

But:

• Revised documents and New Work Items both retain a separate cover page and Table of Contents for the embedded document.
• New Work Items involve Annexes (Form A.1/A.13, and the New Work Item itself): so the embedding document does have different clause numbering to the embedded document. (If Metanorma processes these as a single document, it can't jump back from Annex 1 back to a document preface).
• The Work Item claims to be an Annex, but it has a preface, a title page, and Appendixes, and it is numbered as the source document, not as an annex at all. Which means that calling it an annex has no impact on how it is rendered: it is rendered as a standalone document.

I can tweak the embedded solution to cope with these requirements—automatically extracting a separate cover page somehow, and inserting a table of contents into the embedded document. But the embedding happens as an Asciidoctor include, during document preprocessing. The metadata of the embedded document is preserved in the metadata of the embedding document, as metadata about what the document is based on. But as far as Metanorma processing is concerned, this really is a single document; and telling Metanorma to jump back from annex to preface is going to break its processing model.

That's why the real solution to rendering these has to be a document collection. @ronaldtse is going to be unhappy about that, because document collections are more complicated to set up. I have no sympathy for that objection: document collections are going to be preconfigured in the mn-templates-itu, and I'm not going to destroy the Metanorma processing model to force what is clearly two separate documents to be processed as a single document, but end up looking like two separate documents anyway. These are two separate documents, and Metanorma needs to render one document at a time.

This is going to require added a lot of implementation effort. First, we have never dealt with document collections as a single Word document at all—although making that happen will not be prohibitive. Second, we need directives on what the cover page and intro page of each document looks like in Word, which will be a truncation of the standalone cover and intro pages, passed as custom templates by the collection.

Each Word document is structured as Title section (no page numbers), Preface section (usually Roman number pagination), Main section (Arabic number pagination). The workflow is going to be:

• Title + Preface + Main section of first document
• For each remaining document
  • Custom title document template, if supplied
  • Preface + Main section of that document

By default, each section (preface and main section of each document) will restart page numbering from 1, but I can introduce configuration to continue numbering.

In the case of ITU New Work Items, the custom title document will be:

Annex B

{{docidentifier}}

{{doctitle}}

Summary: {{abstract}}

Keywords: {{keywords}}

In other words, I am not proposing to mark up the embedded document as an annex at all, but to keep the annex as an artefact of the cover page. As I said above, the embedded document is not rendered as a true annex: it really is just treated as a separate document. So I'm not feeling guilty about that kind of subterfuge.

[opoudjis]

@taddhar I am not expecting feedback on these comments, and I am not inviting extensive discussion on it. This is an implementation challenge I need to surmount, and I am laying out the considerations, since any course I take is going to have serious consequences for the structure of the implementation of Metanorma overall. And I cannot keep kludging solutions to embedding autonomous content. This is harder than I'd have preferred, and I'm going to investigate what I can do about it.

[opoudjis]

I'm going to create a separate ticket for the generation of Metanorma collections in Word. Once I've got a satisfactory outcome on that, I'll resume work on this ticket.

[opoudjis]

Resuming work on this, after a lot of distractions.

We have a framework for generating document collections in Word. We now need to set up templates in mn-templates-itu for:

• Generic contributions (already done) (e.g. SG17-C-0223)
• Contributions with revised text (e.g. SG17-C-660): collection, consisting of contribution document + target document
• Contributions with new work item: collection, consisting of contribution document + form as annex + target document as annex
  • Recommendation as target document: A.1 form as annex (e.g. SG17-C-0593)
  • Other document type as target document: A.13 form as annex (SG17-C-0639)

[opoudjis]

There's an issue that the coverpage of the embedded target document has keywords on its coverpage, but standalone documents already have keywords after the table of contents; so there is an inconsistency between the Collection Embedding positioning of keywords (in the ad hoc coverpage) and the standalone document (whose formatting I am not overriding, other than its cover page.)

This is not worth fixing right now, and if anything, I'd argue that the keywords should be removed from the place they've ended up in the embedding cover page.

[opoudjis]

Populating A.1/A.13 from document metadata is raising a problem. Do we author them as:

• Lutaml templates, populated while preprocessing Asciidoc?
• Programmatic Presentation XML text, populated after the Semantic XML is already parsed?

The former allows us to interleave metadata and document content. But if we are going to use document metadata in the form, we have to either duplicate the attribute processing done in Presentation XML (which is a coding smell), or duplicate the document metadata in a YAML data source.

The latter is how we populate cover pages already, including the metadata of the contribution cover page. It gives us more flexibility in formatting, and it lets us reuse parsed data. But it requires us to enter all its information as document metadata (document attributes), not as document text.

For example, we already provide question information in document metadata, like this:

`:question: Q10/17: Identity management and telebiometrics architecture and mechanisms, "Q11/17: Generic technologies (such as Directory, PKI, formal languages, object identifiers) to support secure applications"`

This is broken up into separate questions, and separate identifiers and titles, but that happens in Metanorma XML generation. It has not happened during preprocessing, which only knows about the raw "question" attribute. I don't want to write the parsing of the question attribute in two places, or enter it in two places (the document attribute and some YAML file).

But if I do the forms as programmatically generated text, they have to be fully programmatically generated: I can't do that as half programmatic, half document, like this:

[[annexA]]
[appendix,obligation=normative]
== {% if doctype = "recommendation" %}A.1 justification for proposed draft new Recommendation {{ docnumber }}{%else%}A.13 justification for proposed Draft new {{ doctype }} {{docnumber }} "{{doctitle}}" {%endif%}

|===
h| Question: | {{question}} h| Proposed new ITU-T {{doctype}} | {{ date }}
h| Reference and title: 3| {{ doctitle }}
h| Base text: 2| Annex B of this Contribution h| Timing: | 2020-12
h| Editor(s): 2| {{author }} h| Approval process: | AAP

The "2020-12" and "AAP" have to be document attributes as well. Which is fine, except we also have big chunks of open-ended formatted text in the same table, which does not make sense as document attributes:

5+a| *Scope* (defines the intent or object of the {{ doctype }} and the aspects covered, thereby indicating the limits of its applicability):

This Technical Report provides a structured “Collection of Security Concerns s with their contexts in use cases of Digital Systems, Solutions, Infrastructures and Networks for extracting the Security Requirements to help develop a comprehensive and Granular Cyber Security Reference Architecture as proposed in X.cs-ra.

(It shall develop a common harmonized and standardized approach & Template to describe diverse use cases for their respective context, security concerns & requirements. The output of this work item  shall be highly crucial in developing a comprehensive Cyber Security Reference Architectures by analyzing the collected Use Cases in appropriate context, as well as integrate Cyber Security in a structured manner in various digital systems, solutions, infrastructures & Networks Architectures.)

5+a| *Summary* (provides a brief overview of the purpose and contents of the {{ doctype }}, thus permitting readers to judge its usefulness for their work):
5+a|

The integration of  TCP/IP technology  with traditional protocols , for all activities has converted/transformed the digital Systems & Digital Infrastructures and Networks into a interconnection of Operations Technology(OT) part of the network and information Technology (IT)  Part of the network.  Security risks and concerns are managed differently in IT and OT networks. Security in IT is primarily focused on protecting data confidentiality, ensuring that sensitive information is accessible only to authorized users. OT security, however, prioritizes the safety and availability of industrial systems and processes. The integrity and continuous operation of physical equipment are paramount in OT, given that a failure can result in significant safety and financial implications.

5+a| *Relations to ITU-T Recommendations or to other standards* (approved or under development):
5+a|

. ITU-T X.arch-design
. ITU-T X.800
. ITU-T Z.150
. ISO/IEC/IEEE 42010
. ISO/IEC/IEEE 42020
. ISO/IEC/IEEE 42030
. IEC SRD 63188
. ISO 15704
. ISO/IEC/IEEE 15288
. ISO/IEC/IEEE 12207
. ISO/IEC 31000 Family
. ISO/IEC 27000
. IEC 62559-1, -2, -3 & -.

5+a| *Liaisons with other study groups or with other standards bodies:*
5+a|

. ISO/IEC JTC1/Sc27
. ISO/IEC JTC1/WG13
. ...

5+a| *Supporting members that are committing to contributing actively to the work item:*
5+a|

Ministry of Communications (India)

I can't do this, because the processed metadata like {{question}}, let alone the multiple rows of {{ editor }}, are not available at the time any such Lutaml is generated.

The way out of this is to enter the textual bits of Annex A with identified fixed subheadings, and assemble the table out of a combination of those subheadings and the processed metadata. This is similar to the approach taken in participant lists for IEEE. So:

[[annexA]]
[appendix,obligation=normative]
== Annex A

=== Base text

xxxx

=== Scope

xxxx

=== Summary

xxx

=== Relations to standards

xxx

=== Liaisons

xxx

=== Supporting members

xxx

The output form A.1/A.13 is going to be populated in Presentation XML, as a combination of those labelled subheadings in the Annex A, and the document metadata. It's somewhat clunky, but the alternatives are clunkier.

[opoudjis]

Will allow type on annex, since we will set annex[@type = 'justification'] for the A.1/A.13 form annex.

[opoudjis]

This is done. There is a contribution-new-item and a contribution-revision in mn-templates-itu, both generated as collections, and the latter incorporates A.1/A.13, identified to be populated as [annex,type=justification], and populated from document attributes, and labelled blocks in the annex.