Output NISO STS XML format from an ISO OBP HTML

ronaldtse commented 5 months ago

From:

https://github.com/metanorma/iso-5598/issues/13

The ISO OBP HTML is actually rendered from data of an XML format called "NISO STS" (the ISO flavor of it).

Instead of just the HTML output, we also want to output the NISO XML format.

Use case

Some ISO authors have to start documents from the ISO website as they are unable to obtain the STS files.

On the ISO OBP, informative content, such vocabulary, is freely available. The best way is to give them an automated way to extract this data.

Mechanism

The steps shall be as follows:

Download the index.html for a particular URN
From index.html, convert it into an STS XML document (using the code in the PR #6), and then using the new sts gem to write it as STS XML.

This is a Ruby script that somewhat parses index.html, it's not yet complete. It is provided in:

6

CLI:

$ bundle exec exe/obp-access -o output iso:std:iso:5598:ed-3:v1:en
$ bundle exec exe/obp2sts output/index.html

=> writes out:

output/index.html.xml: STS XML file generated by obp2sts
output/index.html.sts.xml: STS XML file generated by the sts gem given output/index.html.xml as input

Library:

stshtml = Obp::StsHtml.new('output/index.html')
stsxmltext = stshtml.clean.to_xml
sts = Sts::NisoSts::Standard.from_xml(stsxmltext)
puts sts.to_xml(pretty: true)

Work to be done

[ ] obp2sts: Ensure that the StsHtml class completely converts all content from HTML to STS
[ ] obp2sts: Ensure resulting STS file from StsHtml#to_xml is properly parseable by the sts gem (main branch)

roberthopman commented 4 months ago

ruby 3.3.0, branch rt-obp2sts, running bundle exec exe/obp-access -o output iso:std:iso:5598:ed-3:v1:en and bundle exec exe/obp2sts output/index.html, it returns in index.html.sts.xlm one line with <standard/>. At this moment, the expected output? @ronaldtse

ronaldtse commented 3 months ago

@roberthopman sorry for the delay in replying!

one line with <standard/>

No, it is supposed to provide content. Right now, the output is incorrect. The task is to fix the output.

So there are 3 steps:

The input is output/index.html, which is the raw HTML fetched using the obp-access command. It is correct.
The intermediary file is output/index.html.xml, which is "supposed" to be parseable by the sts gem. It does contain content, but is apparently incorrect and hence cannot be parsed by the sts gem.
The final file is output/index.html.sts.xml, which is generated from output/index.html.xml. It is empty because it cannot read the intermediary file.

ronaldtse commented 3 months ago

This is now updated in #6, with now a document structure created using the sts gem.

It already does a reasonable transform of the HTML file into STS by declarative building.

There are a number of TODOs in the code:

The mixed content elements, such as <p> and <sec>, do not fully contain proper content. e.g. if you had <std-id> or <i> inside the content, they will be lost. This is a general issue about Sts::Mapper because I don't know how to actually use it properly. (ping @HassanAkbar )
The "Terms and definitions" section need additional treatment, see the sample document.
The "Normative references" section need treatment like the bibliography, which currently works to some extent (it does build a proper <ref-list> from ISO 5598, but I haven't tested against other documents.
Annexes are not handled right now.
(Important for @HassanAkbar ) I can't get the sts.to_xml method to generate XML content except for <standard .../>. Please help.

We're getting there.

metanorma / obp-access