Implement per-document attachments

[ronaldtse]

Increasingly organizations are issuing PDFs that contain attachments in order to incorporate machine-readable content.

Other than the ISO 10303 series, the BIPM's many NMLs (national metrology laboratories) have begun issuing digital calibration certificates in PDF form with machine-readable attachments, such as XML data.

In the recent APMP DXFG webinar: https://apmp-dxfg.org/dxfg-eoy-webinar-2022.html

Japan's NMIJ provided a preview of their digital calibration certificate that contains multiple attachments:

Metanorma today supports per-collection attachments, but creating a collection is a complex ordeal and requires the user to understand many things (e.g. manifest file, cover page, collection metadata).

We need to support per-document attachments so that users that only deal with one document file can easily incorporate and link to the incorporated attachments as native document elements.

[ronaldtse]

This issue is needed for:

• https://github.com/metanorma/iso-10303-detached-docs/pull/148

[opoudjis]

I really really really want to hand this work over to someone else, so @abunashir, this is my thinking in attachments in general.

In Metanorma Collections, a single attachment can be referenced by multiple files in the collection, so I think the approach I took there is the correct one:

• The attachment is a file that is part of the collection, same as any other file in the collection
• The attachment is referenced as an external file (<eref>), not as a part of the current document (<xref>).
• And external file referencing means a bibliographic entry for the attachment, and a document identifier
  • The document identifier for the attachment is resolved to the file location of the attachment; that is done the same way as for any file in the collection, using a manifest
  • The document identifier for the attachment is flagged as belonging to the collection through a special flag in the bibliography
  • The attachment is flagged as being an attachment, needing special compilation, in the manifest
• The special compilation of the attachment is to compile it as an HTML document if it is ADOC, and to leave it alone if it is TXT. If it is compiled as an HTML document, it is "headless": the initial coverpage and boilerplate of a full Metanorma document is not attached to it.

How to do all this is more or less documented in https://www.metanorma.org/author/topics/document-format/collections/#collection-manifest

There is a long outstanding ticket for you, https://github.com/metanorma/annotated-express/issues/97 , on automating the generation of ADOC for attachments, from a single liquid template, because Ronald was not comfortable incorporating shell scripts into Metanorma generation. I think it is pointless to try to prevent shell scripts, because attachments can come from all sorts of sources and have all sorts of content, automatically or manually generated.

But that's not the concern here.

The concern here is to extend attachment functionality to single documents.

I think much of how I've already implement attachments makes sense, and should be carried over:

• Attachments are external documents
• Attachments are compiled separately
• Attachments are referenced as external documents
• Attachments are signalled in references

The one thing that doesn't apply in the case of single documents is that there is no collection manifest, to indicate which documents are attachments, and which are not. There are two possible ways around this:

1. Convert the single document being compiled into a collection, complete with its own manifest. In fact, I already do this for when we want to compile a single document into an HTML document with one page per section, with the sectionsplit option: https://github.com/metanorma/metanorma/blob/main/lib/metanorma/sectionsplit.rb
2. Do not set in motion an entire collection, but invoke the referencing and compilation of attachments without a manifest

Right now, attachments use the following machinery:

Manifest:

manifest:
  level: brochure
  title: Brochure/Brochure
  docref:
    - fileref: si-brochure-fr.xml
      identifier: si-brochure-fr
    - fileref: attachment.txt
      identifier: ABC
      attachment: true

Bibliography:

[bibliography]
== Bibliography

* [[[theattachment,repo:(current-metanorma-collection/ABC)]]]

Where repo:(current-metanorma-collection/...) instructs metanorma gem to look up the identifier in the manifest

Reference:

<<theattachment,Attachment 1>>

I think we can work around this by doing:

No Manifest

Same Reference

Bibliography:

[bibliography]
== Bibliography

* [[[theattachment,repo:(attachment/{{file path of attachment relative to current file}})]]]

When metanorma sees repo:(attachment/...) links, it inserts a hyperlink to the attachment, just as metanorma gem does right now, but it presupposes the file name remains the same, and just changes the file suffix. And seeing an attachment in the bibliography during processing of Presentation XML triggers compilation of the attachment, if the file suffix warrants it.

The code to do all of this is there, it's just mostly in metanorma gem, with a little bit in metanorma-standoc; this task would involve migrating some of that code to isodoc. If code is invoked in both isodoc and metanorma, I've tended to move it to metanorma-utils.

[abunashir]

Thanks a lot @opoudjis, this is really a very useful information. I'm trying to wrap my head around the whole collection as there are quite some tasks around those and see if we can somehow come up with something that could address most of those common issues.

Both approaches works great, but just to explore the no manifest attachment options, let's say we want to reference attachments inline (in the doc) and and indicate it needs some pre-processing, how would you do that in the asciidoc? is this enough?

* [[[theattachment,repo:(attachment/{{file path of attachment relative to current file}})]]]

@ronaldtse: Do you have any preference, how would you like to include the attachment in documents?

[opoudjis]

* [[[theattachment,repo:(attachment/{{file path of attachment relative to current file}})]]]

I believe that would be enough. Obviously I would need to do some stuff once I see the "attachment" prefix, but because I've already implemented repo:(current-metanorma-collection/...), at least retrieving that indication is going to be straightforward. If the file is cited with an attachment prefix, I will know that it is an attachment, and I will do the required preprocessing...

... using the handles you will provide :))

[abunashir]

That's great, I feel like this could a good approach to investigate and see if we can make that work easily :)

[ronaldtse]

This issue is URGENT. @abunashir what's going on here?

[abunashir]

Hey @ronaldtse, I actually haven't worked on it since a while now, I was prioritising the works on Corado/Oscal, but If it's still urgent then I can start having look into it :)

[abunashir]

I'm looping back to this issue again, @ronaldtse - So, if I understand correctly, we want to support attachment in any document, and it doesn't have to be a collection. Something similar to adding an image in any asciidoc, but it might needs some preprocessing in this case?

I'm also trying to understand what actually needs to happen internally when there is an attachment macro in the asciidoc, let me know if you have any idea, it might save me some time :)

[opoudjis]

Btw, @abunashir, I have given you the code, but I haven't given you one final bit, which you will need to do for ISO-10303.

I have asked you to insert the attachment generating code in metanorma before the sectionsplit code in document generation, which breaks a document into one document per section.

But there will be documents that need to run both sectionsplit, and attachment generation. In fact, ISO-10303 parts are such a document.

Which means you will need to ensure that both attachment generation and sectionsplit can be run on the same document. (You would run attachment generation first.)

[opoudjis]

There are two tasks being described here, and that has confused matters:

• We need to add links to arbitrary binary files from a Metanorma document, and package those files optionally with the XML file as included binaries, or else generate a file manifest for the XML distribution, packaging the Metanorma XML with those binary files.
• For ISO 10303, we have an additional step of compiling and generating the attachments from Metanorma itself.

That second step is a distraction, and I am making it out of scope of this ticket.

The solution is going to remain as I sketched above:

• Asciidoc > Semantic XML

  • Attachments will be given in the bibliography, as * [[[theattachment,attachment:({{file path of attachment relative to current file}})]]]
  • Mime magic will be used to work out what the file is.
  • References to theattachment will resolve to hyperlinks to the given file. (The mechanisms for that are already in place.)
  • If that attribute is off, the attachments will (temporarily) be stored a subfolder for live accessing (with a predictable name like _{document-name}_attachments)
  • Possibly, that subfolder name becomes configurable; but I'm not enthusiastic about that prospect.
  • If a document attribute data-uri-attachment is on (default), the binary files will ALSO be embedded into the document, in the metanorma-extension container, as MIME-encoded binaries.

• Semantic XML > Presentation XML

  • The compilation checks for metanorma-extension/attachments, and if they are present, and the predicted _{document-name}_attachments subfolder is empty, it will export the attachments to that folder
  • References to the attachment are resolved to hyperlinks to the file in the subfolder, and the reference entry for the attachment is deleted.

• Presentation XML > HTML, DOC, PDF

  • The attachments are in the _{document-name}_attachments subfolder, and are linked to in the generated document

• Metanorma gem

  • If the document attribute datauri-attachments is off, then the Semantic XML, Presentation XML, and HTML outputs are zipped up along with the _{document-name}_attachments subfolder at the end of processing.
  • Some sort of file manifest may also be needed.
  • Whatever is consuming metanorma output (site generation?) needs to know when its outputs are zipped files and not raw XML files.

I will do the standoc and isodoc bits of this, and I would want to hand over the metanorma gem part of this to someone else. @alexeymorozov ?

I will likely make the zip + manifest task a new ticket.

[alexeymorozov]

We discussed with Nick, and I'll start as soon as the base work in done.

[opoudjis]

If all bibitems in a references section are hidden, the references section needs to be hidden as well. Debug existing code (it didn't realise that a title could be supplied.)

[opoudjis]

This task is done. I am going to create a new task for @alexeymorozov and for @Intelligent2013

[opoudjis]

Will document attachments, and sidestep the zipping work, since it may have been overtaken by capsium.